Skip to main content

Full text of "Legislative oversight and program evaluation: a seminar sponsored by Congressional Research Service"

See other formats


94th Congress 1 
2d Session / 



COMMITTEE PRINT 



LEGISLATIVE OVERSIGHT AND 
PROGRAM EVALUATION 



A SEMINAR SPONSORED BY 
CONGRESSIONAL RESEARCH SERVICE 

PREPARED FOR THE 

SUBCOMMITTEE ON OVERSIGHT PROCEDURES 

OF THE 

COMMITTEE ON GOVERNMENT OPERATIONS 
UNITED STATES SENATE 

BY THE 

CONGRESSIONAL RESEARCH SERVICE 



LIBRARY OF CONGRESS 







MAY 1976 



Printed for the use of the Committee on Government Operations 



^ SSET } COMMITTEE PRINT 



LEGISLATIVE OVERSIGHT AND 
PROGRAM EVALUATION 



A SEMINAR SPONSORED BY 
CONGRESSIONAL RESEARCH SERVICE 

PREPARED FOR THE 

SUBCOMMITTEE ON OVERSIGHT PROCEDURES 

OF THE 

COMMITTEE ON GOVERNMENT OPERATIONS 
UNITED STATES SENATE 

BY THE 

CONGRESSIONAL RESEARCH SERVICE 
LIBRARY OF CONGRESS 




MAY 1976 



Printed for the use of the Committee on Government Operations 



U.S. GOVERNMENT PRINTING OFFICE 
71-867 WASHINGTON : 1976 



COMMITTEE ON GOVERNMENT OPERATIONS 
ABRAHAM RIBICOFF, Connecticut, Chairman 
JOHN L. McCLELLAN, Arkansas CHARLES H. PERCY, Illinois 

HENRY M. JACKSON, Washington JACOB K. JAVITS, New York 

EDMUND S. MUSKIE, Maine WILLIAM V. ROTH, Jr., Delaware 

LEE METCALF, Montana BILL BROCK, Tennessee 

JAMES B. ALLEN, Alabama LOWELL P. WEICKER, Jr., Connecticut 

LAWTON CHILES, Florida 
SAM NUNN, Georgia 
JOHN GLENN, Ohio 

Richard A. Wegman, Chief Counsel and Staff Director 

Paul Hoff, Counsel 

Paul L. Leventhal, Counsel 

Eli E. Nobleman, Counsel 

David R. Schaefer, Counsel 

Matthew Schneider, Counsel 

John B. Childers, Chief Counsel to the Minority 

Brian Conboy, Special Counsel to the Minority 

Marilyn A. Harris, Chief Clerk 

Elizabeth A. Preast, Assistant Chief Clerk 

Harold C. Anderson, Staff Editor 



Subcommittee on Oversight Procedures 
SAM NUNN, Georgia, Chairman 
HENRY M. JACKSON. Washington JACOB K. JAVITS, New York 

EDMUND S. MUSKIE, Maine CHARLES H. PERCY, Illinois 

JAMES B. ALLEN. Alabama WILLIAM V. ROTH, Jr., Delaware 

LAWTON CHILES, Florida 

W. P. Goodwin. Jr., Chief Counsel and Staff Director 

Rochelle Joxes, Professional Staff Member 

Merrielol Howser, Chief Clerk 

Gary Klein, Minority Counsel 

<n) 




'HJtnUeb £bi<\ie& Jfoenale °-"=— «• 




May 19, 1976 



Honorable Abe Rlbicoff 

Chairman 

Committee on Government Operations 

United States Senate 

Washington, D.C. 20510 

Dear Mr. Chairman: 

On behalf of the Subcommittee on Oversight Procedures, 
I hereby submit a report, "Legislative Oversight and 
Program Evaluation: Proceedings of a Seminar Sponsored 
by the Congressional Research Service, May, 1975." 

This report was compiled by the Congressional Research 
Service, Library of Congress, at my request. It contains 
papers and other materials prepared in connection with a 
seminar sponsored by CRS in May of 1975 at the initiation 
of Senators Lee Metcalf and Bill Brock. 

The seminar was designed to acquaint congressional staff 
with the various techniques of oversight and the resources 
available to them. Special emphasis was given to program 
review and evaluation. 

The Subcommittee on Oversight Procedures has a 
responsibility to develop more effective methods of overseeing 
the Executive branch of the government. The proceedings of 
this seminar should contribute to a better understanding of 
oversight methods on the part of congressional staff and, 
consequently, should lead to more effective oversight. 

On behalf of the Subcommittee, I respectfully request 
that this report be printed for the use of the Government 
Operations Committee in order to make the materials available 



(iii) 



IV 



to Members of Congress and staff members who were not able 
to attend the seminar. 




Introduction 1 

Summary of the Main Themes of the Seminar Series on "Legislative 

Oversight and Evaluation," May 1975 9 

Day I: Overview of Responsibilities and Resources for 

Legislative Oversight and Program Evaluation: Summary of 

Proceedings 28 

"Introductory Remarks," Lester S. Jayson, Director, 

Congressional Research Service 41 

"Congressional Oversight: A Review of Recent Legislative 
Activities," Walter J. Oleszek, Congressional Research 
Service 44 

"Program Evaluation Concepts," Genevieve J. Knezo, 

Congressional Research Service 55 

"Ask the Right Questions," Dennis L. Little, Futures Group, 

Congressional Research Service 63 

Discussion 70 

"Congressional Research Service: Resources for Oversight and 
Evaluation," Norman Beckman, Deputy Director, Congressional 
Research Service (Dr. Beckman is now Acting Director) ... 71 

"The Service's Responsibilities for Emerging Issues and 
Terminating Programs," David E. Gushee, Congressional 
Research Service 90 

"Monitoring Reports to Congress Required by Law," 

Ellen C. Collier, Congressional Research Service 94 

Discussion 104 

["The General Accounting Office's Responsibilities for 

Oversight and Evaluation in Reference to the Congressional 
Budget and Impoundment Control Act"], -James P. Oliver, Deputy Director 
Financial and General Management Studies Division, General 
Accounting Office 10 5 

Discussion 113 

(v) 



VI 



TABLE OF CONTENTS (Cont.) 



Page 



"The Office of Technology Assessment: A Description of 
the Office's Objectives and Activities," Gary Thomas, 
Office of Technology Assessment 114 

Discussion 145 

["Oversight Responsibilities and Activities of the Senate 
Committee on Government Operations, and Several of Its 
Subcommittees"] Marilyn Harris, Chief Clerk; Howard J.' 
Feldman, Chief Counsel, Permanent Subcommittee on 
Investigations; James Davidson, Professional Staff Member, 
Subcommittee on Intergovernmental Relations; Victor 0. 
Reinemier, Staff Director, Subcommittee on Reports, 
Accounting and Management; Ron Chiodo, Professional Staff 
Member Subcommittee on Federal Spending Practices, Efficiency, 
and Open Government, and Bill Goodwin, Jr., Staff Director, 
Subcommittee on Oversight Procedures 146 

Day II: Congressional Committee Case Studies of Evaluation and 

Oversight: Summary of Proceedings of Day II 164 

"Foreign Affairs Oversight: Role of the Staff Survey Mission," 
John H. Sullivan, Staff Consultant, House Committee on 
International Relations 173 

Discussion 186 

"The Freedom of Information Act," William G. Philips, former 
Staff Director Foreign Operations and Government Information 
Subcommittee, House Committee on Government Operations . . . 188 

Discussion 205 

"Congressional Committee Case Studies: Studies in Public 

Welfare, " Alair A. Townsend, former Research Director of the 
Subcommittee on Fiscal Policy, Joint Economic Co mmi ttee . . 206 

Discussion ." 223 

"Policy Formation Through Program Evaluation and Systems 

Analysis: A Congressional View," James W. Giltmier, Senate 
Committee on Agriculture and Forestry 224 

Discussion 243 



VII 

TABLE OF CONTENTS (Cont.) 



Page 



Day III: Executive Branch Evaluations and the Processes of 

Evaluation: Summary of Proceedings 245 

"Federal Program Evaluation: The Perspective from OMB," a 

paper prepared for the American Political Science Association 
by Clifford W. Graves and Stefan A. Halper, used as a basis 
for remarks presented by Clifford W. Graves, Deputy 
Associate Director for Evaluation and Program Implementation, 
Associate Director for Management and Operations, Office of 
Management and Budget 258 

Discussion i 275 

"Current Evaluation Activities in the General Accounting 

Office," Keith E. Marvin, Associate Director, Financial and 
General Management Studies Division, General Accounting 
Office 277 

Discussion 287 

"Practical Application of Evaluation in Legislative Processes," 
William H. Foskett, as consultant to Sen. Bill Brock .... 288 

Discussion 302 

"The State of the Art of Program Evaluation," Clark Abt, 

President, ABT Associates 303 

"Evaluating Education Programs Are We Getting Anywhere?" 

Article prepared by John W. Evans, used as a basis for remarks 
by John W. Evans, Assistant Associate Commissioner, Office 
of Education 333 

Discussion 340 

"Evaluating Evaluation: A Congressional Perspective," Allen 

Schick, Congressional Research Service 341 



VIII 



APPENDIX 






A. Oversight Plans of the Committees of the U.S. House of Representatives 

A report by the Committee on Government Operations, 1975^ 355 

B. Overview of Federal Agency Program Evaluation Activities: Joint 

Inquiry by the Office of Management and Budget and the General 
Accounting Office* 436 

C. Selected General Accounting Office Reports Involving Program 

Evaluation 457 

D. Program Evaluation: A Manual for Legislators and Legislative Staffs, 

Draft of January 1975, prepared by William H. Foskett, and 
Harrison W. Fox, Jr., for Sen. Bill Brock, (Tennessee) 464 

E. Program Evaluation: Emerging Issues of Possible Legislative Concern 

Relating to the Conduct and Use of Evaluation in the Congress 
( and the Executive Branch, Multilith, Congressional Research 
'Service, November 16, 1974r " 499 

F. Evaluations of Manpower Training Programs: Selected References, 

1970-1974, Multilith, Congressional Research Service, 

October 25, 1974. --583 

G. Evaluation Research in Public Administration: Selected References, 

1967-1974, Multilith, Congressional Research Service, October 21, 
1974. 593 

H. Evaluation Research in Social Policy: Selected References, 1970- 

1974, Multilith, Congressional Research Service, October 31, 1974 

603 

I. Evaluation Research and Its Application to Health Programs: Selected 
References, 1970-1974, Multilith, Congressional Research Service, 
February 24, 1975, 615 

J. Evaluation of Public Welfare Programs: Selected References, 1970^ 

1974, Multilith, Congressional Research Service, April 8, 1975.633 




THE LIBRARY OF CONGRESS 

Congressional Research Service 



WASHINGTON. D.C. 205*0 



The Honorable 

Sam Nunn 

Chairman, Subcommittee on Oversight 

Procedures 
Committee on Government Operations 
United States Senate 
Carroll Arms Hotel -- Room 502 
Washington, D. C. 20510 

Dear Mr. Chairman: 

I am pleased to transmit this material which you requested us 
to prepare for possible publication as a committee print. The document, 
"Legislative Oversight and Evaluation: Proceedings of a Seminar Spon- 
sored by the Congressional Research Service, May 1975," contains the 
proceedings of the seminar series held on May 5, 12, and 19, 1975. 

As you will recall, the series was requested by Senator Lee 
Metcalf, Chairman of the Government Operations Subcommittee on Reports, 
Accounting, and Management and Vice Chairman of the Joint Committee on 
Congressional Operations, and by Senator Bill Brock, Ranking Minority 
Member of the Subcommittee on Reports, Accounting, and Management of 
the Government Operations Committee. 

Following the seminar, we received your request to prepare 
the proceedings for possible publication. You noted that "...the 
Subcommittee is planning to publish a series of documents and prints 
on oversight and evaluation for legislative staff orientation and 
reference purposes. The publication of the seminar series proceedings 
in committee print form would be a valuable contribution to the Sub- 
committee's program of improving information and knowledge about leg- 
islative oversight and evaluation." 

As requested by your staff we have prepared a brief summary 
of the main themes of the seminar series. A more detailed summary of 
each of the three sessions precedes the actual papeis and discussion 
for that session. We have also included appendix materials that 
speakers distributed at the meetings to illustrate their oversight 
and evaluation activities. 

We should like to acknowledge the participation of the many 
persons who cooperated with us in presenting the seminar, delivering 
papers, and preparing the proceedings. The Oversight Procedures Sub- 
committee staff and the staff of the Subcommittee on Reports, Accounting 

(ix) 



and Management deserve special thanks. We should also like to acknow- 
ledge the assistance given by the speakers for their work in preparing 
and delivering their presentations and in some cases editing them for 
publication. Representatives of the General Accounting Office also merit 
particular mention for their assistance in planning the seminar as well 
as taking part in it. 

Staff members of several Divisions of the Congressional Research 
Service -- in particular, Genevieve J. Knezo of the Science Policy Research 
Division -- cooperated in developing the agenda for the meetings. Gary 
Evans of the CRS Office of Special Programs was the coordinator for the 
seminar series. The proceedings were assembled and summarized by Ms. Knezo 
with the cooperation of Mr. Evans and of members of the Science Policy 
Research Division. The project as a whole was carried out under the 
joint direction of myself and Warren Johnston, Chief of the CRS Office 
of Special Programs. 

We have been pleased to undertake this project. We hope the 
proceedings will be of value to you and your staff. 



Sincerely, 




fbrman Beckman 
Acting Director 



INTRODUCTION 

A. Purpose and Scope of the Seminar 

On May 5, 12, and 19, 1975, the Congressional Research Service 
sponsored a three-day seminar series on legislative oversight and pro- 
gram evaluation. The Service held the series at the request of Senator 
Lee Metcalf, Chairman of the Senate Government Operations Subcommittee 
on Reports, Accounting, and Management, and Senator Bill Brock, Ranking 
Minority Member of that Subcommittee. The series had two purposes. 
The first was to acquaint legislative staff who have responsibilities 
or interest in legislative oversight and program evaluation with 
current requirements, processes, and resources for oversight and evalua- 
tion. The second was to provide a forum for discussing problems that 
staff might encounter in conducting oversight and in doing or using 
evaluation research. 

Speakers at the meetings consisted of committee staff; members 
of the staff of the General Accounting Office (GAO), Congressional 
Research Service (CRS), and Office of Technology Assessment (OTA); 
officials from the Office of Management and Budget and the Office of 
Education; and nongovernmental specialists. 

An average of 75 persons attended each of the three all afternoon 
sessions. They were committee and member staff aides, staff of congressional 
support agencies such as the CRS, GAO, and OTA, and staff of the executive 
branch and nongovernmental evaluation research communities. 

Shortly after the series ended, Senator Sam Nunn, Chairman of 
the Subcommittee on Oversight Procedures of the Senate Committee on 
Government Operations asked the CRS to prepare the proceedings of 



the seminar for publication. The two objectives of the publication 
would be: (1) to summarize the major issues and preserve the pro- 
ceedings for the benefit of those legislative staff unable to at- 
tend the meetings, and (2) to compile a comprehensive resource docu- 
ment on the topic of oversight and evaluation. 1 / 

The proceedings consist of this introduction; a summary of major 
themes raised during the series; the papers presented each day, pre- 
ceded by a summary of the day's proceedings; and appendix materials 
consisting of documents speakers used to illustrate their activities. 

The papers are presented in the order in which they were given. 
The meeting was not recorded; therefore highlights of discussions which 
follow the papers are based on notes taken during the meetings. 

B. The Context of Evaluation and Oversight 

The current period of increasing budgetary restraints and com- 
peting claims on scarce Federal resources , coupled with an increasing 
proliferation of Federal benefit programs, has created a climate conducive 
to the adoption of better methods of program planning, management, 
and oversight. During recent years the agencies of the executive 
branch have instituted several important reforms to improve the ef- 
ficiency and efficacy of Federal spending. These include both modi- 
fied and new administrative arrangements, and development of new 



_l/ Senator Nunn indicated that this document would be one part of a 
series that the Subcommittee on Oversight Procedures expected to 
publish on the topic of improving resources and techniques of 
oversight. Another volume planned in the series would include 
analytical essays on oversight and evaluation. 



decisionmaking and policy analysis techniques. Technical reforms are 
seen in the adoption of such techniques as: Management by Objectives 
(MBO) , program evaluation, cost/benefit analysis, and cost/effective- 
ness analysis. However, these new techniques are constrained by some 
of the same limitations which plagued their 1960s precursors, PPBS 
and systems analysis. These include: (1) unrealistic expectations 
of analytical tools and techniques, whose imprecision and relatively 
unsophisticated level of development hamper the generation of wholly 
valid and reliable information; (2) inadequacies of data collection 
and generation procedures which may vitiate the utility of tight 
analytical and conceptual schemes; and (3) imprecise understanding 
of the contribution of quantitative analytical information to gov- 
ernmental and political decisionmaking. 

During 1972, Senator William V. Roth of Delaware undertook a 
study of Federal evaluation practices. The conclusion of this study 
underscored the need for better analysis and evaluation of Federal 
programs, and, at the same time, identified many of the problems which 
hamper effective analysis and evaluation. One of the paragraphs of 
the study summarizes, for both decisionmakers and analysts, the lim- 
itations and ambiguities of much analysis: 

The use of analytical techniques is subject to a number of dan- 
gerous distortions. These include over-objectif ication, over- 
systemization, and use for advocacy by program managers and 
political executives. We must keep in mind that it is especial- 
ly difficult to gauge whether social programs are successful. 
These programs necessarily have multiple goals which in their 
ultimate form are very hard to measure. Further I think we 



need to guard against the erection of complicated formal struc- 
tures of analysis which have no impact on decisionmaking. 2/ 

Subsequently, in commenting on this study as well as on its own 
commissioned studies in the area, the Joint Economic Committee identi- 
fied another problem which hampers the utility of quantitative evalu- 
ation and analysis: that is the fact that political decisionmaking 
in a democracy must include a number of different and sometimes con- 
flicting types of information. Systematically generated analytical 
information is only one type of information politicians use in making 
decisions: 

Analysis can provide an important and helpful tool for making 
decisions, but it is no more than a tool. Problems involving 
social policy and value judgments must be considered and weighed 
in conjunction with the results of benefit-cost analysis, and 
the final decision made by the human policymaker. 3 / 

Despite these limitations, attempts to improve the conduct and 
utility of analysis and evaluation have continued unabated in recent 
years. This is evidenced by several types of events. First, aca- 
demics, extramural policy researchers and analysts, and evaluators, 
either singly or in conjunction with governmental officials, have 

continued to discuss the issues raising the level of awareness 

and debate about the conduct and utility of quantitative analytical 



2 / Report on A Survey of Federal Program Evaluation Practices. By 
Hon. William V. Roth, Jr., a U.S. Senator from the State of Dela- 
ware, pp. 1-52 in U.S. Congress, Joint Economic Committee, Bene - 
fit-cost Analysis of Federal Programs . A compendium of papers 
submitted to the Subcommittee on Priorities and Economy in Gov- 
ernment. 9 2d Cong., 2d sess. January 2, 1973, Washington, U.S. 
Govt. Print. Off., 1973, p. quoted on p. 53. (Joint committee 
print . ) 

3 / Benefit-Cost Analysis of Federal Programs, op. cit . , p. 54. 



techniques, especially those which are based on the methods of the 
social sciences. 4 / 

Second, the executive and legislative branches' principal 
fiscal oversight agencies, the Office of Management and Budget and 
the General Accounting Office, have considerably increased their ef- 
forts to improve the management, conduct, utility, and understanding 
of the analysis and evaluation research enterprise. Both of these 
agencies, as indicated in the discussions below, have undertaken 
a number of activities to develop policies to govern the evaluation 
research process; to improve legislative language for evaluation, 
and to forge better links between evaluation and budgeting. GAO in 
addition has special responsibilities for evaluation under the Congres- 
sional Budget and Impoundment Control Act of 1974, P.L. 93-344. 



_4_/ There are numerous documents on this subject. Some of the more 
recent and general treatments are: 

(1) Beckman, Norman and Harold Handerson, eds. New Directions in 
Public Administration. Proceedings of "Opposing Forces in Public 
Administration," The Fifth Annual Conference of the National Cap- 
ital Area Chapter of the American Society for Public Administra- 
tion, November 14-15, 1974. Reston, Virginia, The Bureaucrat, 
Inc., 1975: Part II. "Program Evaluation: Does It Improve Pub- 
lic Policies and Administration?", pp. 58-98. 

(2) Lyons, Gene M. , ed. Social Research and Public Policies. The 
Dartmouth /OECD Conference. Dartmouth, The Public Affairs Center, 
1975. 206 p. See especially Part II, Social Experiments, pp. 87- 
140 and "Part III, Evaluation Research," pp. 141-205. 

(3) Knezo, Genevieve J. Program Evaluation: Emerging Issues of 
Possible Legislative Concern Relating to the Conduct and Use of 
Evaluation in the Congress and the Executive Branch. Multilith, 
Science Policy Research Division, Congressional Research Service, 
Library of Congress, Nov. 16, 1974. 79 p. (75-35 SP.) 



Third, and undoubtedly most important, the Congress itself has adopted 
several recent measures to improve the conduct of evaluation and oversight 
and to better link program planning, budgeting, evaluation, and oversight. 
These activities include: (1) passage of the Legislative Reorganization Act 
of 1970, which strengthened the policy analysis and information capabilities 
of the Congressional Research Service, called upon the General Accounting 
Office to make available to all members and committees of the House and 
Senate lists of GAO reports and when requested the reports themselves, and 
required the preparation of biennial committee reviews of legislative 
activities; (2) passage of the House Committee Reform Amendments of 1974, 
H. Res. 988, which enlarged oversight responsibilities of standing committees, 
especially of the House Government Operations Committee; and (3) passage of 
the Congressional Budget and Impoundment Control Act of 1974, P.L. 93-344, 
which authorized committees to conduct or make use of evaluation and 
augmented GAO's capability to aid committees in conducting evaluation 
and in writing requirements and legislative language for evaluation. 
Other related activities include: creation of the House Information Systems 
Group, which provides members and staff with access to models and data 
useful for evaluation purposes; creation of the Office of Technology 
Assessment, to provide the Congress with unbiased information and analysis 
regarding the positive and negative impacts of technological applications; 
creation of functionally distinct oversight committees in the Senate 
Committee on Government Operations; and a study of the need to revamp 
the Senate committee structure. Informal activities along these lines, 
including several sponsored by Senator Bill Brock, consist of a monthly 
seminar for legislative staff on program evaluation, and preparation of 
a draft manual on program evaluation concepts and techniques to assist 



legislative staff who have responsibilities for writing requirements for 
evaluation into new legislation. 

As recently as five years ago, evaluation units in the Federal Gov- 
ernment were rare. Today they are widespread, __5_/ and by the latest 
OMB account, Federal agencies, exclusive of the Departments of De- 
fense and Treasury, are responsible for spending almost $120 million 
annually for evaluation research studies. 6 / Evaluation research is 
a relatively new management and policy research tool, widely used, but 
as evidenced in the CRS seminar and in other forums, beset with the 
problems which accompany any new decisionmaking technique. These 
problems cannot be solved without considerable effort by both 
evaluation performers and users. In concluding the November 1974 Amer- 
ican Society for Public Administration seminar on evaluation research, 
Keith E. Marvin of the General Accounting Office noted in this respect: 
". . .Despite the continued existence of some basic problems and ques- 
tions concerning evaluation work, there will be more such activity at 
both the agency and the legislative staff level in the future. What is 
accomplished will become at least as important as how well." ]_/ 



5/ Lafayett, David P., et. al. "Federal Department and Agency Ex- 
periences Using Program Evaluation." In Beckman, New Directions 
in Public Administration: The Federal View, op_. cit . , p. 69. 

_6_/ U.S. Office of Management and Budget. Evaluation and Program 
Implementation Division. Survey of Federal Evaluation Activi- 
ties, Sept. 1975. Washington, U.S. Govt. Print. Off., 1975. 246 p. 

Jj "Program Evaluation: Does It Improve Public Policies and Admin- 
istration: Summary of Panel and Floor Discussions." In Beckman, 
New Directions in Public Administration: The Federal View, op_. 
cit., p. 97. 



The CRS seminar on Legislative Oversight and Program Evaluation 
was designed to provide a forum for identifying and discussing some 
of* the major issues surrounding evaluation and oversight, especially 
in relation to the responsibilities of legislative staff. The seminar 
addressed topics of both production and implementation, such as method- 
ology, data, funding level, procurement, type of contractor, training, 
timing of evaluation, intelligibility of reports, and use of qualita- 
tive and quantitative evaluation information in legislative decision- 
making and oversight. Neither the summary which follows nor the semi- 
nar were designed to resolve issues regarding evaluation and oversight, 
but to raise the level of awareness and debate on the issues as a modest 
step toward improving the efficiency and efficacy of Government. 



SUMMARY OF THE MAIN THEMES OF THE SEMINAR SERIES ON "LEGISLATIVE 
OVERSIGHT AND PROGRAM EVALUATION," MAY 1975 



A. The Scope and Agenda of the CRS Seminar on Legislative Oversight and 
Program Evaluation 

Each seminar session was organized around a separate theme. The 
first afternoon's presentations overviewed general responsibilities 
and resources for oversight and evaluation. House and Senate Govern- 
ment Operations Committee staff, as well as staff from GAO, CRS, and 
OTA, made presentations and assisted in answering questions. 

Congressional committee experiences in conducting evaluations or 
using secondary policy analyses or evaluations done by congressional 
support agencies, or quantitative analytical evaluation research re- 
ports performed by or for executive agencies , formed the theme of the 
second seminar session. The substantive issues addressed were over- 
sight and evaluation of foreign policy, by the House Committee on 
International Relations; implementation of the Freedom of Information 
Act by the Subcommittee on Foreign Operations and Government Infor- 
mation of the House Committee on Government Operations; examination 
of the Joint Economic Committee's extensive investigations and 
studies of public welfare; and committee evaluation and oversight by 
the Senate Committee on Agriculture and Forestry of the Farmers Home 
Administration and other agricultural programs. 

The third seminar session had a double theme: (1) the state-of-the- 
art of evaluation and (2) executive branch evaluation activities. 
Speakers addressed the pros and cons of various evaluation research 
methods; efforts to enhance oversight of federally funded evaluation 
research; and efforts of the GAO and Office of Management and Budget 



10 



to refine executive agency evaluation units and processes. Two members 
of Senator Brock's staff described their monthly evaluation seminar for 
legislative staff and their manual on evaluation, for legislative users. 
The agenda also included a case study of evaluation in the Office of 
Education, and a presentation by an expert on evaluation research methodology. 

B. Introduction to Oversight 

Walter Oleszek of the Congressional Research Service defined over- 
sight as the exceedingly complex process of determining whether executive 
agencies carry out programs as the Congress intended. He said that in 
the past, the Congress had experienced difficulties in conducting effective 
oversight. These difficulties were attributable to several factors, in- 
cluding: 

- a lack of staff with time for oversight, since most staff efforts 
had to be used to write new legislation; 

- a tendency to avoid formal in-depth oversight because of the 
advocacy relationships which might have developed between some committees 
and the agencies they oversee; and 

- a lack of established procedures for cooperation between Government 
Operations Committees and authorizing committees in sharing oversight 
information. 8 / Other speakers also identified potential obstacles to 
effective evaluation. These included: a reluctance on the part of some 
Federal agencies to doing evaluation research whose results might ultimately 



_8_/ Oleszek, Walter. "Congressional Oversight, A Review of Recent 
Legislative Actions," Day 1. 



11 



affect agency budgets and staff size; _9_/ and difficulties both in 
determining which evaluation techniques most appropriately answer over- 
sight questions and in pinpointing policy options that evaluation and 
oversight studies might generate. 10 / 

C. Types of Oversight 

There was an implied consensus about several different types of 
oversight and techniques which can be used to support these activities. 
Walter Oleszek differentiated among three types of oversight. "Legisla- 
tive oversight" is done by authorizing committees to determine whether 
agencies have properly implemented programs under their jurisdiction. 
"Fiscal oversight," primarily a function of the Appropriations Committees, 
scrutinizes Federal spending. "Investigative oversight," until recently 
a responsibility primarily of the House and Senate Government Operations 
Committees, examines programs which cut across agency and congressional 
committee jurisdictions. A committee staff member added definitions for 
other more specific and "operational" types of congressional oversight. 
"Routine oversight" consists of daily monitoring of Federal agency regula- 
tions to determine whether the agency is following the intent of the law. 



9 / See especially papers and discussions of: Giltmier, James W. "Policy 
Formation Through Program Evaluation and Systems Analysis: A Congres- 
sional View," Day 2; and Schick, Allen, "Evaluating Evaluation: A 
Congressional Perspective," Day 3. 

10 / See especially Knezo, Genevieve J., "Program Evaluation Concepts." 
On policy options related to evaluation, especially the need for 
"decision-driven evaluation," see Graves, Clifford W. , remarks based 
on Federal Program Evaluation: The Perspective From 0MB," Day 3; 
and Abt, Clark C. "The State of the Art of Program Evaluation," 
Day 3." 



12 



"Special oversight," like investigative oversight, involves special 
in-depth investigation of a high priority area which might result in 
the need for corrective legislative action. "Guerilla oversight" consists 
of spot-checking, in a one day hearing, where adversary proceedings can 
lead to corrective action by an agency, but without legislative enactment 
or amendment. 11 / 

D. Types of Evaluation Research 

Federal agency administrators generally conduct or contract for one 
of two types of quantitative evaluation studies: process evaluations or 
impact evaluations. These are done at the request of the Congress or 
for the agency's own management purposes. These types of evaluation are 
used for different purposes and are grounded in different types of techniques, 
Process evaluations are used to answer management questions. Generally 
process evaluations use non-experimental evaluation techniques and data 
generated in the normal course of administering and monitoring a program. 
These types of evaluation are far less costly and require far less time 
to do than do impact evaluations. Impact evaluations answer questions 
about whether a program caused a change in attitude or behavior in the 
participants or targets of a program. Impact evaluations are more 
difficult to do than process evaluations. Generally they require 
experimental research techniques and randomized test and control groups. 



11 / Remarks of Phillips, William G. "The Freedom of Information Act, 
Day 2. 



13 



New and costly data collection procedures must be built into a program, 
usually in the program authorization stage, to meet the requirements of 
impact evaluation. 12 / 

E. Scope and Limitations of Evaluation Research for Legislative Oversight 

It was noted that impact and process evaluation are useful for many 
types of legislative oversight — but most especially for those types 
dealing with program formulation and amendment. 13 / However, most 
speakers seemed to agree that information generated using any one specific 
evaluation technique constitutes only one type of information that members 
use in making political decisions. On this point, Clark Abt, emphasized: 
" What we cannot do with program evaluation is create new programs, make 
decisions, make the assumption of rationality come true, and get users 
or consumers of program evaluations to ask the right questions. We also 
cannot make value judgments that have any credibility where these are 
decisive program decisions." 14 / 

However, the point was made repeatedly that many evaluation research 
studies are done improperly, as a result they fail to yield information 
useful for decisionmaking. Several factors contribute to this situation. 



12 / See Knezo, op. cit . Abt, op. cit. , elaborated upon these themes. 

13 / Foskett, William H. "Practical Application of Evaluation in Leg- 
islative Processes," Day 3. 

14 / Abt, op. cit. , Day 3. 



14 



First, the need to represent different political viewpoints usually 
produces broad or ambiguous statements of legislative program goals. As 
a result evaluators have only imprecise yardsticks against which to 
measure program or agency accomplishments. 15 / 

Second, evaluation researchers and program managers often lack 
awareness of the questions that evaluation and oversight can and cannot 
answer and the proper techniques to use to answer these questions. 
Legislative staff are not always aware of these issues. 16 / 

In relation to this issue, Allen Schick suggested that the Congress 
should be wary of getting directly into the business of carrying out 
program evaluations. His rationale was that legislative oversight is not 
analogous to program oversight, which requires specialized evaluation 
techniques and should be a responsibility primarily of the Federal agencies 
which administer programs. 17 / 

Several legislative staff speakers, however, emphasized the need 
for legislative program evaluation. A few gave illustrations of committees 
conducting, monitoring, or using evaluations. They pointed out that Title 
Seven of the Congressional Budget and Impoundment Control Act, P.L. 93-344, 
authorizes committees to conduct, contract for, or request executive 
agencies to do evaluations. It was also noted that executive agencies 
are increasingly calling for the conduct of evaluation research. A recent 



15 / See especially remarks of Little, Dennis L. "Ask the Right Questions,' 
Day 1, and Abt, and Oleszek, op. cit. 

16 / See presentations of Foskett, op. cit ., and Abt, op. cit. , especially. 

17 / Remarks of Schick, op. cit. , Day 3. 



15 



General Accounting Office study, for instance, demonstrated the federally 
funded program evaluation research studies cost at least $130 million in 
the Fiscal Year 1974. It was also noted that because of the Budgeting Act 
and increased oversight responsibilities in the House, committees are 
increasingly writing legislative requirements for program evaluation into 
bills authorizing new programs — requiring staff to be intimately familiar 
with the promises and perils, methods, and utility of different types of 
program evaluation research studies. 

There was general agreement that the Congress has limited resources, 
cannot and does not desire to compete with or duplicate the information 
gathering and analysis capabilities of the executive branch, and has 
oversight information requirements different from the information require- 
ments of Federal executive agencies which administer programs. Therefore, 
committee staff with responsibilities for oversight and evaluation might 
be more selective in: 

- determining priorities for oversight and evaluation, and 

- using their own staff or the staff of the auxiliary legislative 
information and analysis support bodies, as well as the staff of Federal 
agencies, to review existing literature to determine if there already is 
information available to assist in developing priorities for evaluation 
and oversight. 

F. Illustrations of Committee Oversight and Evaluation Activities 

The case studies of oversight and evaluation presented during the 
seminar illustrated a variety of qualitative and quantitative information 
gathering and analysis techniques available to committee and requirements 
for their successful use. 



16 



Dr. Alair Townsend, a former staff member of the Subcommittee on 
Fiscal Policy of the Joint Economic Committee discussed a particularly 
comprehensive oversight activity. In its role as an investigative 
committee serving both the House and the Senate, the Joint Economic 
Committee conducted a series of studies and hearings to assess both 
process (management) and impact (program effectiveness) issues relating 
to the Family Assistance Plan. To do this adequately, however, the Com- 
mittee had to assess the Family Assistance Plan in the context of all 
income assistance programs. A series of committee prints dealing with 
these topics was prepared by in-house staff, outside contractors, staff 
of Federal agencies, and' legislative support agencies, such as the 
Congressional Research Service and the General Accounting Office. 

Several important observations were made about requirements for 
the conduct of studies of this nature. First among these was the need 
for a highly trained committee staff to monitor closely the work of 
congressional support agencies and outside contractors. This is 
necessary to keep the work on schedule, and, perhaps more important to 
insure that it conforms to a legislative decisionmaker's needs. This 
type of monitoring is indispensable in the case of certain outside 
contractors, for example those social scientists who tend to use Federal 
research monies more to train students and to produce basic or applied 
research valued by their peers than to produce research useful to policy- 
makers. Close monitoring of outside contractors is required also to insure 
that reports are not too laden with academic jargon. 18 / 



18 / See remarks and discussion following the presentation of Townsend, 
Alair A. "Congressional Committee Case Studies: Studies in Pub- 
lic Welfare," Day 2. 



17 



John H. Sullivan Illustrated another unique legislative oversight 
device in his presentation describing some of the oversight activities 
of the House Committee on International Relations and the Senate Committee 
on Foreign Relations. These committees use overseas field staff visits 
by highly trained committee staff investigators. One of the principal 
advantages of this method of information gathering is that it generates 
information that can rarely be obtained any other way. It is also one 
of the most direct means of supplying oversight information because 
research is done by staff members who have the trust, confidence, contact, 
and rapport with members required to insure ready access to policymaking. 
However, effective overseas field visits also require that staff be highly 
trained and conduct visits in pairs to corroborate verbal evidence which 
executive officials might deny later if called to testify. 19/ 

Various other oversight and evaluation techniques were described in 
the case studies presented during the second day of the seminar. These 
included the conduct of program evaluation research by staff of the 
Senate Committee on Agriculture and Forestry, the use of public opinions 
polls by the Senate Committee on Government Operations to obtain informa- 
tion for investigative oversight in the area of intergovernmental relations, 
and the relevance to congressional oversight of the subpoena power, as 
described by staff of the Senate Committee on Government Operations. 



19 / See remarks and discussions of presentation by Sullivan, John H. 
"Foreign Affairs Oversight," Day 2. 



18 



G. Use of Congressional Support Agencies 

Several illustrations were given of instances where committees used 
a variety of support services provided by the Congressional Research 
Service, the General Accounting Office and the Office of Technology 
Assessment to assist in oversight and evaluation. They included, for 
example, assessment by the CRS of the quality and extent of use of 
executive branch reports to Congress as required in foreign affairs 
legislation; the use of the CRS to analyze agency responses to a committee 
questionnaire on implementation of the Freedom of Information Act; and 
the use of the Service in preparing analytical reports on the Family 
Assistance Program. Because of its long history in doing auditing and 
oversight work for the Congress, the General Accounting Office was fre- 
quently cited as a useful information and analysis source. Instances 
of GAO assistance ranged from illustrations of its aid in overseas 
investigations to assistance in evaluating the Farmers Home Administration. 

It was indicated, however, that there are constraints on the use of 
these legislative support resources similar to the limitations on the use 
of other outside evaluation resources. These constraints consist primarily 
of the need for close committee staff monitoring to insure that research 
is done in terms and in time-frame relevant to decisionmakers. 20 / 

1. Office of Technology Assessment . Dr. Gary Thomas of the Office 
of Technology Assessment noted that the OTA has adopted a policy of 
interfacing closely with committee staff, not only to receive suggestions 
for assessment topics but also to help design research appropriate to 



20 / See papers of end discussions following the papers of Giltmier, 
op. cit. , and Townsend, op. cit. 



19 



legislative needs. The OTA ensures this interface by providing requesting 
committees with periodic, sometimes monthly, progress reports from con- 
tractors during the course of a technology assessment to assist if necessary 
in meeting any immediate legislative decisionmaking needs. Also in support 
of this objective the OTA has begun to do mini-assessments or exploratory 
assessments in addition to the longer-range studies it usually funds. 
Mini-assessments have either one of two purposes: (1) to determine whether 
a more in-depth assessment may indeed be required; or (2) to do a short 
two-week inhouse study to meet members' immediate information needs. Such 
a study was done recently to, assist members in reviewing the budget of the 
Energy Research and Development Agency, ERDA. 21 / 

2. Congressional Research Service . The basic authority for the 
Congressional Research Service to support legislative oversight and 
program evaluation comes from the Legislative Reorganization Act of 1970. 
The CRS provides several different types of support to fulfill its respon- 
sibilities. Among those described at the seminar were: policy analysis 
reports, reports dealing with terminating issues and emerging issues; 
seminars on important public policy issues; the preparation of issue briefs; 
formation of a futures group; and refinement of automated information 
systems capabilities including bibliographic citations by subject, bill 



21 / See presentation of, and discussion following, Thomas, Gary, "The 
Office of Technology Assessment: A Description of the Office's 
Objectives and Activities," Day 1. 



20 



digests, access to technical information and bibliographic data bases 
including MEDLARS, MEDLINE, and legal codes and econometric models. 22 / 

3. The General Accounting Office . The General Accounting Office 
recently instituted several activities intended to improve its oversight 
and evaluation support capabilities. These include, for example, plans 
to establish a Congressional Information Services Group to coordinate 
and help standardize social program information required for evaluation 
and to improve Congress 1 and GAO's capabilities to do and use program 
evaluations as required under the provisions of P.L. 93-344. In addition, 
the agency is preparing an inventory of evaluation studies done for 
Federal agencies as well as developing methodological criteria and standards, 
and legislative language formats for committees to assist them in making 
the tradeoffs required between different types of evaluation strategies 
in relation to specific evaluation decisionmaking needs. 23 / 

Some discussion was given to potential limitations on the use of 
the General Accounting Office in assisting with legislative evaluation 
and oversight. First, it was noted that Federal agency administrators 
tend to be less cooperative and informative with GAO staff than with 
committee staff. This occurs, one staff person said,, because Federal 



22 / See the following papers: Beckman, Norman, "Congressional 
Research Service: Resources for Oversight and Evaluation," 
Day 1; Gushee, David E. "The Service's Responsibilities for 
Emerging Issues and Terminating Programs," Day 1; and Collier, 
Ellen C. "Monitoring Reports to Congress Required by Law," Day 1. 

23 / See presentations and discussions following: Oliver, James P. 
"Remarks on the General Accounting Office," Day 1; and Marvin, 
Keith E., "Current Evaluation Activities in the General Accounting 
Office," Day 3. GAO's report on methodological standards was 
published as: U.S. General Accounting Office. Exposure Draft: 
Evaluation and Analysis to Support Decisionmaking. December 9, 1975 
52 p. (OPA076-9). 



21 



administrators attribute less power to GAO staff than to committee staff. 
It was also noted that sometimes there is a long "turnaround" time for 
the completion of some GAO reports, which may take a year to complete 
and often require a time-consuming seven-step review process. In response, 
agency and legislative staff noted, the GAO has begun to institute informal 
reporting procedures which reduce the review time and provide staff with 
copies of draft reports. However, it was also noted that staff sometimes 
use these reports as if they were the final version of a study, thereby 
jeopardizing the integrity of the oversight process. 24 / It was also 
pointed out that some GAO reports customarily result from statutory re- 
quirements enacted in the past. As a result the GAO might not always 
have resources available to prepare reports on new issues currently before 
the Congress. 25/ 

H. Evaluation in Executive Agencies 

Several of the speakers described problems in the conduct and use 
of Federally funded evaluation studies. For instance, one committee 
staff person noted that Federal agencies frequently refuse to release 
evaluation reports to the Congress. It was also remarked that some 
agency administrators tend to avoid objective evaluation because they 
are the captive of their programs and would like to believe that pro- 
grams are indeed benefiting clients. Several speakers noted that often 
in the past neither administrators nor some members of Congress wanted 
evidence to demonstrate that a program might be ineffective. The 



24 / Discussions and papers, Day 2, especially Giltmier, op. cit. 
25/ Idem. 



22 



gravity of this situation is compounded by the fact that most evaluations 
are done on new, innovative social programs, not on long established pro- 
grams which may profit more from good evaluation, but which may be more 
costly in terms of possible bureaucratic retrenchment. 26 / 

Some discussion focused on the issue of properly institutionalizing 
the locus of performance of evaluation within an agency to enhance its 
use. One speaker suggested that agencies should separate evaluation from 
budgeting, because agencies usually try to maximize their budgets and 
retain programs. Therefore, the possibly useful adversarial nature of 
some evaluation is submerged to other interests. This speaker also 
suggested that evaluation might be performed best in a separate inspector 
general office within an agency due to its critical adversary nature. 27 / 
However, others submitted variants on these recommendations. For instance, 
one staff person suggested that evaluation would be used more effectively 
if it were sponsored by an agency attached to the Office of the Secretary 
of a Department. John Evans described a variant used in the Office of 
Education: the project manager for an evaluation reads the study, analyzes 
its policy implications, and sends a memo to OMB, via the Commissioner 
of Education, identifying the policy implications. 28 / Several speakers 
agreed in principle with this comment, noting that agencies and legislative 



26 / See presentations and discussions if any, following the presentations 
of Giltmier, op. cit ., Day 1, Phillips, op. cit ., Day 2, Oleszek, 
op. cit ., Day 1, and Schick, op. cit ., Day 3. 

27 / Schick, op. cit ., Day 3. 

2fl / See discussion following and paper of Evans, John, Remarks based on 
"Evaluating Education Programs — Are We Getting Anywhere?" Day 3. 



23 



staff should make a special effort to assure the conduct of "decision- 
driven" evaluations, which are designed specifically to generate infor- 
mation for policymaking. 29 / All generally agreed that agencies should 
strive to improve the development of more useful evaluation report dis- 
semination mechanisms, both within agencies and to the public. 

With respect to the development of better policies to govern 
evaluation research, it was noted that the GAO and the OMB are doing 
a joint study to collect information about evaluation research in the 
executive branch, including an inventory of funding for evaluation, 
purposes of the evaluation report, and identification of the offices 
supporting such work. The OMB is also undertaking several activities 
to improve the conduct of evaluations within agencies. These include 
an assessment of the uses of agency evaluations, an assessment of 
evaluation training needs, and publication of a technical manual on 
evaluation management systems. 30 / 

I. Methodological Needs 

Clark Abt raised several points about how the Congress and agencies 
might improve the methodological sophistication of evaluation research. 
In summary, these were: more training for evaluation researchers; Federal 
sponsorship of more cross-cutting evaluations, which attempt to assess 
program impacts in a variety of geographic locations and demographic 



29 / See especially statements of Graves, op. eit ., Day 3 and Abt, 
op. cit . , Day 3 . 

30/ An illustration of one OMB report of this nature is: Survey of 
Federal Evaluation Activities. By Evaluation and Program 
Implementation Division, Office of Management and Budget. September 
1975. 246 p. 



71-867 O - 76 - 4 



24 



groups; better Federal agency preparation of requests for proposals 
(RFPs) ; and improved Federal specification of program goals, policy 
options, or information needs that evaluation can support. 31 / 

J. Evaluation Requirements 

Clifford Graves, Deputy Associate Director, Evaluation and Program 
Implementation Division of the Office of Management and Budget, summed 
up in his oral remarks the difficulties the Congress faces in using or 
contracting for evaluations. He noted that an OMB assessment of Federal 
agency evaluations indicated that evaluations generally are untimely, 
inaccurate, and irrelevant for decisionmaking. 32 / 

In this respect, several speakers, including executive officials 
and contractors as well as committee staff people who have used evalu- 
ations or who have been instrumental in heightening legislative staff 
awareness of the scope and limitations of program evaluation, outlined 
points that congressional staff might consider when doing or using 
evaluations or writing legislative language requiring evaluation for 
oversight purposes. These points may be summarized as follows. Staff 
might consider the needs to: 

- Set priorities for doing evaluations . In addition, staff or 
contractors should be encouraged to review existing evaluation literature 
to see if studies have already been done on the same or related programs. 



31/ Abt, op. cit. , Day 3. 
32 / Graves, op. cit ., Day 3. 



25 



Doing so is necessary to help formulate questions, to review the state 
of the art of evaluation methods, and to determine what kind of evalua- 
tion is needed. It also helps to encourage fiscal responsibility. Staff 
might also seek evaluations of past program enactments, which consume con- 
siderable Federal expenditures, as opposed only to new program enact- 
ments. 

- Develop appropriate statements of agency or program goals to 
assist in developing language for later evaluation efforts . This may 
be a potentially troublesome problem, since legislators frequently use 
vague or ambiguous statements of program objectives for the sake of 
political compromise. Sometimes vague statements of program goals 
hamper effective evaluation because evaluators are left with imprecise 
yardsticks against which to compare actual program effectiveness and 
accomplishment . 

- Determine whether a process or an impact evaluation is required . 
This is an important consideration that must account for the following 
factors: what are the questions which decisionmakers want answered 
about the program? Are they mainly questions regarding program manage- 
ment, or are they questions regarding the actual effects or impacts of 
the program on its intended objects? When do the decisionmakers need 
these answers? Are decisionmakers willing to endure the possible 
political side-effects of impact evaluations or experimental evaluations 
required to answer cause and effect questions? Tradeoffs must be made 
among all these factors. 

- Be aware of the possible policy options available to legislators 
which might result from evaluations, and write legislative language for 
evaluation which is designed specifically to support the generation of 



26 



information and analysis relevant to those policy options which are 
available. It was noted in this respect that the General Accounting 
Office is now surveying Federal statutes which require evaluation to 
develop standards adequate for generating appropriate legislative 
language for evaluation. 

- Determine the amount of lead-time available for evaluation in 
relation to possible decisionmaking options . It should be remembered 
that more time is required for the conduct of complex, longitudinal, 
quantitative evaluations of ongoing programs than for qualitative or 
non-longitudinal evaluations. 

- Recognize that impact evaluations usually cannot employ data 
collected in the normal course of monitoring a program . If impact 
evaluations are required, the legislative language for evaluation might 
include the requirement of developing appropriate new data systems, as 
well as authorization for funding of these systems. However, staff 
should be aware of the possible disadvantages of such data collection 
procedures, including invasion of privacy and the ethical Issues of 
experimentation with human beings. 

- Monitor very closely the conduct of evaluation , whether by 
inhouse staff, a congressional support agency, or an executive branch 
agency, to insure that evaluation reports generate information and 
analysis to answer the questions that legislative decisionmakers want 
answered about the program. 

- Develop appropriate ways to disseminate information to members 
and require that summary evaluations reports for members be readable, 
short, and free of jargon. 



27 



- Increase efforts to insure that evaluation Information is used 
in authorization and appropriations processes and in Federal agency 
decisionmaking processes. 



28 



DAY I: OVERVIEW OF RESPONSIBILITIES AND RESOURCES FOR LEGISLATIVE OVERSIGHT 
AND PROGRAM EVALUATION: SUMMARY OF PROCEEDINGS 



The material presented at the first day's seminar on the topic of 
general responsibilities and resources for legislative oversight and 
program evaluation set the theme and tone for succeeding discussions. 
In summary, oversight and evaluation do not consist of one immutable 
process applicable to all legislative decisionmaking needs. The pur- 
poses, techniques, and resources of oversight and evaluation differ con- 
siderably depending upon a decisionmaker's needs, and related to these, 
tradeoffs staff have to make among available evaluation resources, types 
of programsbeing assessed, and lead-time available for information gather- 
ing and assessment. However, it was stressed during the first day's 
meeting and throughout the seminar that executive and legislative staff 
sometimes do not give enough attention to determining how particular evalua- 
tion techniques can generate information appropriate to decisionmaking 
needs, especially with respect to evaluations mandated by the Congress. 
This lack of attention can result in needless and wasted research expenditu: 

The first day's meeting was divided into two parts: first, respon- 
sibilities and needs for oversight and evaluation; and second, resources 
for oversight and evaluation. 

A. Legislative Responsibilities and Needs for Oversight and Evaluation 

Mr. Lester S. Jayson, Director of the Congressional Research Service, 
opened the meeting by describing the importance to the legislative pro- 
cess of an exploratory seminar on oversight and evaluation . Then he 
described the agenda for the series, indicating that statements would be 
made by executive branch officials, evaluation contractors, and legisla- 
tive staff, from both committee and congressional information and analysis 
support agencies. 



29 



1. Scope and Limitations of Legislative Oversight. Walter Oleszek , 
of the Government Division of the Congressional Research Service, 
presented the keynote paper on the first part of the seminar theme, 
legislative oversight. He defined oversight as the exceedingly complex 
process of determining whether executive agencies carry out programs 

as the Congress intended. Three types of oversight can be distinguished: 
legislative, fiscal, and investigative. 

Mr. Oleszek noted that the Congress has encountered innumerable 
problems in doing oversight. Chief among these have been (1) imprecise def- 
inition of programs goals, (2) lack of adequate staff, (3) lack of time 
required to do adequate oversight (since most staff time is used to 
prepare new legislation), (4) non-objectivity resulting from the 
mutually reinforcing relationships that sometimes develop between 
legislative committees and Federal agencies, and (5) lack of established 
patterns of cooperation among authorizing, appropriations, and government 
operations committees in sharing information relevant to agency and program 
oversight. 

The speaker also described recent legislative activities designed 
to improve committee organization and information use and gave a cautiously 
optimistic assessment of their implications for improving legislative 
oversight. These actions include: the Legislative Reorganization Act 
of 1970, the Congressional Budget and Impoundment Control Act of 1974, 
P.L. 93-344, and House passage of the Committee Reform Amendements of 
1974, H.Res. 988. 

2. Types of program Evaluation . The keynote for the second part 

of the seminar theme — program evaluation was given by Genevieve 

Knezo of the Science Policy Research Division, Congressional Research 
Service. Ms. Knezo noted the increasing demand for and proliferation 



30 



of evaluation studies as well as the growing dissatisfaction that 
members of Congress, evaluation researchers, and agency administrators 
are expressing regarding the use of evaluation in policymaking, includ- 
ing legislative oversight. Many of these inadequacies, she noted, might 
be attributed to misunderstanding by legislative staff and agency admin- 
istrators about the techniques used and products generated by different 
types of evaluation. 

Two types of evaluation research are done for Federal agencies: 
process evaluations and impact evaluations. Process evaluations; are used 
to answer management questions; generally they use non-experimental 
evaluation techniques and data gathered in the normal course of adminis- 
stering and monitoring a program. They usually are not as costly as the 
second type of evaluation: impact evaluation. Impact evaluations 
usually answer questions about the impact of given programs, i.e., did a 
program cause the change expected in attitude or behavior in 
participants? Impact evaluations are difficult to do; generally they 
require both experimental techniques with randomly Selected test and con- 
trol groups, and also use of new data collection techniques which have to 
be built into a program even before it is funded. 

Ms. Knezo also introduced a theme which was heard several times 

during the seminar series: sometimes the results of evaluation, whether 

a well-conducted evaluation or a poorly conducted evaluation, or whether 

favorable or unfavorable to a program, are not used in the oversight 

process. This failure occurs because the legislative process requires 

that decisions be based on objective information as well as value 

preferences. Sometimes adoption of even a small-scale experimental 

program reflects a political consensus about the feasibility and c 

desirability of a program that no amount of evaluation, whether pro or con 
will alter. 



31 



3. The Need For Dnderstandlng Goals . Dennis Little, 
head of CRS Futures Group, addressed the theme of the need for 
proper specification of goals. He noted: "In order to conduct a mean- 
ingful agency and program evaluation, we need to ask the right questions 
at the right time, and our inquiry must take place at several levels. 
The right questions can be asked only if the evaluator understands the 
agency's goals, missions, program, program objectives, and program and 
administrative evaluation criteria. In most cases such an understand- 
ing does not exist and, consequently, many evaluations are nonexistent 
or of questionable utility." 

He then provided a framework and step-by-step procedure for in- 
ducing agency administrators to give clear statements of their goals, program 
objectives in support of these goals, information needed to assess achievement 
of the objectives or goals, and ways to determine if new evaluation 
studies were indeed required. He used the case of the Social Security 
Administration as an example. 

B. Legislative Resources for Oversight and Evaluation 

The second part of the first afternoon's presentations consisted of 
statements about legislative resources for evaluation and oversight. 
Staff of the Congressional Research Service, the General Accounting Of- 
fice, and the Office of Technology Assessment described how their primary 
and secondary data gathering and analysis activities could serve to assist 
legislative evaluation users. The afternoon concluded with statements 
from staff members of the House and Senate Government Operations commit- 
tees, describing their new responsiblities . 



32 



1. Resources of the Congressional Research Service. Dr . Norman 
Beckman, as deputy director of the Congressional Research Service, be- 
gan these proceedings by providing a general description of the origin, 

evolution, organization, and functions of the Congressional Research 
Service as a legislative support agency. He described several types 
of CRS activities which assist the Congress directly in performing over- 
sight and evaluation. He also gave examples of CRS policy analysis re- 
ports which contributed directly to legislative oversight and evaluation 
functions, including reports relating to mineral leasing; redeployment of 
U.S. armed forces in certain countries; implementation of revenue shar- 
ing; community development; education and recreation legislation; 
retirement systems; and immigration. 

The basic authority for the CRS to support oversight and evaluation 
is the Legislative Reorganization Act of 1970, P.L. 91-510. Several of 
the Act's provisions relate specifically to oversight and evaluation. For 
instance, the act charges the Service with responsibility for: (1) policy 
analysis dealing with new legislative proposals as well as oversight 
programs; (2) new anticipatory functions, including the submission to 
each committee of subject and terminating program lists at the beginning 
of each new Congress; and (3) maintaining continuous liaison with all 
committees. 

Dr. Beckman also detailed innovative directions CRS is taking to 
fulfill its responsibilities. These include holding public policy sem- 
inars for members and staff, which permit expert speakers to discuss all 
sides of an issue, and improvement and enlargement of CRS automated 
information services. These information systems contain data bases 
directly supportive of oversight and evaluation: bill digests and status 



33 



reports; abstracted bibliographic information; and issue briefs which 
provide in a few pages (in print or on videoscreen) the history, current 
status and major bibliographic references on important issues. The 
automated information system also provides researchers with access to 
other special data bases, such as economic models; unpublished economic, 
social, and educational census data; special Federal agency bibliographic 
data bases, such as ERIC, MEDLARS, and MEDLINE: and legal codes. 

a. Emerging issues and terminating reports : Dr. Beckman noted that 
the Legislative Reorganization Act of 1970 gave the CRS new responsibili- 
ties for assisting committee oversight by providing committees, at the 
beginning of each Congress, with lists of important emerging issues and 
of all programs and activities falling under each committee's juris- 
diction which are scheduled to terminate in that Congress. 

David E. Gushee of the Environmental Policy Division, described 
the procedures CRS uses to generate and transmit such lists and the 
form the lists take. A prerequisite for doing the job, he noted, is 
the formation of interdivisional CRS task forces to coordinate the prepara- 
tion of emerging issues lists for committees with jurisdiction over 
several different types of issues. Preparation of the emerging issues 
lists also requires special current awareness by CRS analysts and good 
working relationships with committee staff. He reported that "For the 
94th Congress, the Service organized task forces for 49 committees. The 
lists, when transmitted, all included some oversight topics, and a number 
of them included agreement* on studies the CRS would either do or partici- 
pate in as components of the committee's oversight activities." 

For the 94th Congress, the Service identified 421 expiring programs 
over half were authorizations of appropriations; the next largest 



34 



categories were reports from agencies and commissions and "general 
authority." 

b. Oversight of foreign affairs reports : Often legislation con- 
tains language requiring Federal agencies to report annually or semi- 
annually to the Congress, generating information to be used for over- 
sight purposes. The Foreign Affairs Division of the Congressional 
Research Service assessed the utility and products of these reporting 
requirements in the foreign affairs field jointly for the Senate Committee 
on Foreign Relations and the House Committee on International Relations 
(formerly called the Committee on Foreign Affairs). Mrs. Ellen Collier 
of the Foreign Affairs Division described the work in this area. 

The first report done for the committees identified several in- 
adequacies of reporting requirements: a variety of different and hard- 
to-use reporting forms were used; some reports lacked dates for the 
reporting period covered, or date of publication; in several cases 
statutory requirements led to overlapping reporting requirements; many 
reports were submitted too late; and most of the reports were superficial, 
glossy public relations documents inferior in quality to informational 
or analytical reports needed for good oversight. 

In its first report to the Committees, the Foreign Affairs Division 
observed that legislative use of reports for evaluation and oversight 
would be enhanced if steps were taken to improve the quality of reports 
and if a central information receiving and monitoring system were 
established. 

About a year later the Division prepared a second report for the 
committees evaluating Federal agency responses to the recommendations 
of the first report. Although the Division found improvements, which 



35 



are generating better information for the committees, it noted that far 
more work is needed to improve reporting requirements and compliance with 
them. At the request of the two committees, the Division is now monitoring 
and analyzing several hundred reports which are submitted to the Congress 
by executive branch agencies each year in accordance with about 350 
separate statutory requirements in foreign affairs legislation. 

2. Resources of the General Accounting Office. James P. Oliver, 
of the General Accounting Office, made the next presentation. He gave 
a succinct statement, illustrated with numerous charts and graphs, ex- 
plaining new legislative budgeting requirements mandated by the Congres- 
sional Budget and Impoundment Control Act of 1974, P.L. 93-344. He noted, 
in particular, that properly timed evaluation was needed to fulfill this 
act and to assist authorizing committees in justifying program requests 
as required under the act. He also briefly described the relationships 
between the Office of Management and Budget, the Congressional Budget 
Office, the General Accounting Office, and the House and Senate Budget 
Committees in fulfilling the provisions of the law. 

The General Accounting Office plans to establish a special 
Congressional Information Services Group to enable the agency to fulfill 
its responsibilities under the act for improving and coordinating program 
information and program evaluation techniques and reports . The group has 
responsibility for the following tasks: development of information 
structures and evaluation methods; improvement of reporting to the Con- 
gress of information and evaluations; cooperation with the executive 
branch in improving information systems and evaluation processes; improve- 
ment of congressional and GAO capabilities to acquire and use informa- 
tion and programs evaluations; and cooperation with States and local 
governments in improving the flow of evaluation information. 



3. Resources of the office of Technology Assessment. Dr. Gary 
Thomas, a staff member of the Office of Technology Assessment working 
on materials assessments, presented the next paper, which placed the 
OTA assessment activities in the context of congressional oversight and 
evaluation responsibilities. The speaker described the origins and current 
projects being undertaken by the OTA at the request of the Congress. 

Dr. Thomas also discussed two issues of special importance to the 
oversight and evaluation role played by a legislative support 
agency. He noted that both its legislative mandate and legislative 
exigencies require the OTA's work to be especially timely and responsive 
to the Congress. The requirement is designed, in part, to overcome an 
inadequacy described throughout the meeting: that evaluation reports 
done by Federal agencies and the GAO frequently are of little use to 
legislative decisionmaking needs because they are too late. 

In this same respect, Dr. Thomas noted that the OTA does long- as 
well as short-range assessments. Short range assessments consist of 

two types those done to explore the feasibility and need for a more 

in-depth study and those done on an urgent basis for a particular com- 
mittee. 

Dr. Thomas also noted that OTA maintains very close liaison with 

committees both to receive suggestions for assessments, and to brief 

committee staff throughout the process of the assessent, making avail- 
able working documents and monthly reports contractors submit. 

4. Oversight and Evaluation in the Senate Government Operations 
Committee. The next set of presentations consisted of discussions 
by staff members of the Senate and Hou6e Government Operations Committees 
describing their responsibilities for oversight and evaluation. 



37 



Ms. Marilyn Harris, chief clerk of the Senate Committee on Govern- 
ment Operations, led the discussion by committee staff members. She 
noted that the committee has oversight responsibilities for all Federal 
agencies. To fulfill these responsibilities adequately, the Committee 
recently reorganized, creating a functionally and financially autonomous 
subcommittee structure. She also described the effectiveness of informa- 
tion exchange and cooperation among the subcommittees, and between the 
Government Operations Committee and other standing committees with 
legislative authority. 

The Permanent Subcommittee on Investigations, which does not have 
legislative authority, conducts investigations on topics ranging from 
crime to the Russian grain sale. Howard Feldman, Subcommittee General 
Counsel, noted two especially important oversight information gathering 
tools: the use of the subpena power and of highly trained professional in- 
vestigators. He also noted that the reports of the Subcommittee usually 
lead to the initiation of legislative action by other committees with 
legislative jurisdiction for the issue. 

The Intergovernmental Relations Subcommittee has legislative over- 
sight responsibilities for the relationships between Federal, State, 
and local government officials in areas dealing, for instance, with 
Federal aid programs, questions of constitutional authority, and the 
Freedom of Information Act. The Subcommittee Counsel, Jim Davidson, 
described investigations dealing with revenue sharing, the impact of 
the energy crisis on State and local government, and the New Federalism. 
This Subcommittee had made effective use of another important oversight 
information gathering tool: the use of a Harris public opinion poll to 
obtain citizen's views about the performance of State and local 



38 



govermental officials. Current oversight activities include implementa- 
tion by Federal agencies of the Freedom of Information Act and of activi- 
ties of U.S. intelligence gathering agencies. 

A new subcommittee, the Oversight Procedures Subcommittee, was created 
during the 94th Congress. Its general objectives are to assist other Senate 
committees in improving the oversight function and to anticipate problems re- 
quiring investigation before they become major issues. Its four specific 
responsibilities are: (1) to help formulate better methods and procedures 
for oversight; (2) to oversee the use and disposal of Federal property; 
(3) to assist in developing new budget evaluation measures; and (A) to 
oversee the work of the Paperwork Commission. 

Bill Goodwin, Subcommittee Staff Director, gave several examples of 
current activities to improve oversight and evaluation. These include study- 
ing the Senate committee structure to consolidate jurisdictional responsibili- 
ties and working with the Congressional Research Service and the General 
Accounting Office to prepare and publish- materials on oversight and evalua- 
tion procedures and resources. 

The Staff Director of the Reports, Accounting, and Management 
Subcommittee, Vic Reinemer, gave details on the need for and procedures 
used in two current oversight areas. First is an investigation of over - 
legislation. The committee believes that the legislative process would 
be improved considerably if efforts were made to review and revise existing 
statutes rather than adopting new legislation which may duplicate or counter- 
act existing laws. The Subcommittee is using the services of the General 
Accounting Office and the Congressional Research Service to carry out 
this investigation. 



39 



The second area of investigation is oversight of the Federal Ad- 
visory Committee Act. The committee's preliminary research indicates 
that the Congress and executive agencies are creating far too many ad- 
visory committees; and that existing advisory committees can assume 
whatever new advisory functions may be required. The Congressional 
Research Service and the General Accounting Office are also being used 
for data gathering and analysis on this oversight question. 

The next speaker was a Counsel to the Subcommittee on Federal 
Spending Practices, Efficiency, and Open Government, Ron Chiodo. The 
Subcommittee's jurisdiction extends to Federal procurement laws and 
regulations, efficiency, and economy of Federal spending practices 
generally, and the open conduct of government meetings, including those 
f the Coneress Mr " Cniodo 8 ave details on the objectives and procedures used for 

several current oversight investigations, including: cost overruns of 
military R and D, due to lack of competitive procurement procedures; the 
need to consolidate Federal procurement regulations; oversight of pro- 
cedures to review and award grants and contracts in the Department of 
Health, Education, and Welfare; and "Sunshine" legislation. 

5. Oversight responsibilities of the House Government Operations 
Committee. Under the provisions of H. Res. 988, the Committee Reform 
Amendments of 1974, each committee and subcommittee of the House is re- 
quired to provide the House Committee on Government Operations with a 
plan for its oversight activities at the beginning of each Congress. 
The objective of this requirement is to enhance oversight of Federal 
programs which fall under the Committee's jurisdication and to attempt 
to coordinate oversight activities of the House committees. The 



71-867 O - 76 



40 



Government Operations Committee also has special investigative over- 
sight responsibilities. 

Bill Jones, Staff Director of the Committee, described the 
purposes, methods used, and expected utility to the House of the over- 
sight coordination work the committee did at the beginning of the 94th 
Congress. Committee staff, from both the majority and minority sides 
polled members and staff of all House committees to obtain information 
on the committee's oversight plans. These were later published in a 
committee print which is included as an appendix to this report. 

Mr. Jones said that the document's major purpose is to disseminate 
information about oversight responsibilities and to prompt committees 
to develop long-range oversight plans. There are evidences of overlap 
in oversight jurisdiction; however, according to Mr. Jones, the Govern- 
ment Operations Committee does not now have a responsibility to force 
resolution of overlaping jurisdictions. Instead the committee encourages 
the separate committees and subcommittees involved to resolve differences 
or hold joint investigations or hearings. 



41 



'INTRODUCTORY REMARKS", MR. LESTER S. JAYSON, 
DIRECTOR, CONGRESSIONAL RESEARCH SERVICE 



I welcome you today to the first of three afternoon sessions in a 
seminar series on legislative oversight and program evaluation. The se- 
ries is being sponsored by the Congressional Research Service at the re- 
quest of Senator Lee Metcalf, Chairman of the Senate Government Operations 
Subcommittee on Reports, Accounting, and Management, and Senator Bill 
Brock, Ranking Minority Member of that subcommittee. The purpose of the 
seminars is to acquaint legislative staff who have responsibilities or 
interest in legislative oversight and program evaluation with current 
requirements, processes, and resources. 

The seminars are designed to bring together useful information on 
the various resources available to Congress or assistance in the over- 
sight and evaluation process, and to present a variety of actual experi- 
ences with oversight and evaluation in both the Congress and the Executive. 
The sessions are meant to be informal and exploratory rather than an ex- 
haustive inventory or assessment of legislative oversight responsibilities 
and techniques. Active participation by the audience is essential to the 
success of this series, and I encourage you to raise questions and offer 
constructive comments in the discussion period that follows each presen- 
tation. 

The speakers for today's session will outline new legislative over-- 
sight and evaluation responsibilities and the conceptual and methodolo- 
gical links between these two functions. Staff from three organizations 
which have legislative support functions — CRS, GAO, and OTA — will 
describe some of their resources and activities in support of oversight 



42 



and evaluation, using case studies as examples where appropriate. Repre- 
sentatives of the House and Senate Government Operations Committees will 
then outline their perspectives on committee oversight responsibilities 
and describe related activities. 

On the second day, May 12, four committee case studies representing 
different substantive areas and approaches to legislative oversight and 
evaluation are scheduled for presentation. These case studies should 
help illustrate how different types of evaluation strategies are required 
for effective oversight and assessment of different program areas. Each 
of these strategies, presumably, has distinct advantages and limitations — 
conceptual, methodological, and political — that must be taken into con- 
sideration. 

The third session on May 19 is designed to explore executive branch 
experience with program evaluation and the evaluation process itself. 
The General Accounting Office and the authors of a program evaluation man- 
ual for legislative staff will discuss evaluation processes and appropri- 
ate legislative language and techniques in relation to legislative program 
goals. The Office of Management and Budget will describe the steps being 
taken by OMB to improve the conduct and use of evaluation in the executive 
branch. Dr. Clark Abt, who has extensive experience directing federally 
supported program evaluation research studies, will give an overview of 
the state of the art of evaluation techniques and an assessment of the 
strengths and weaknesses of different approaches. Dr. John Evans of the 
Office of Education will discuss practical issues and problems with re- 
spect to evaluation of education programs, particularly as they relate to 
the legislative process and to congressional interests. 






43 



A more detailed outline of the second and third seminar sessions is 
available for your information and for circulation to others you think 
may be interested. 

In addition, relevant CRS and other background materials will be on 
the table by the door at each session for your information and use. These 
include bibliographies on evaluation, a report on emerging issues relating 
to evaluation, and the House Government Operations Committee report on 
the oversight plans of the House committees. 

Since I will not be able to be present at all sessions, I have asked 
Norman Beckman, Deputy Director of CRS, to be the moderator of the sem- 
inar series. 

Now I would like to introduce the first group of speakers: Mr. Walter 
Oleszek of our Government Division, Mr. Dennis Little of the CRS Futures 
Group, and Ms. Jean Knezo of the Science Policy Research Division. 



44 



'CONGRESSIONAL OVERSIGHT: A REVIEW OF RECENT LEGISLATIVE ACTIVITIES," 
WALTER J. OLESZEK, CONGRESSIONAL RESEARCH SERVICE 



While Members and scholars have long stated that oversight of 
Federal agencies and programs is a principal and traditional function 
of Congress, it was not until 1946 that Section 136 of the Legisla- 
tive Reorganization Act (60 Stat. 132) formally required Congress's 
standing committees to exercise "continuous watchfulness" of the 
execution of laws by administrative agencies under their jurisdic- 
tion. While always considered an important function, Members today 
exhibit heightened concern about oversight. Before discussing why 
there is increased concern with oversight, it might be helpful to 
define the term. 

What is Oversight? 

The 1946 Legislative Reorganization Act divided Congress' over- 
sight function among three types of standing committees. First, author- 
izing committees (Agriculture, Veterans' Affairs, Commerce, and the 
like) principally had "legislative" oversight over programs and agen- 
cies under their jurisdiction. These committees were to assess whether 
programs were working or not, and propose legislative remedies to re- 
solve whatever program deficiencies they might have uncovered. 

"Fiscal" oversight was the primary concern of the Appropriations 
Committees of each chamber, which were to scrutinize the spending of 
funds by Federal agencies. Finally, the 1946 LRA granted wide-ranging 
"investigative" oversight to the House and Senate Committee on Govern- 
ment Operations. With their authority to study the "operation of Gov- 
ernment activities at all levels with a view to determining its 



45 



economy and efficiency," the Government Operations Committees were 
authorized to examine the administration and execution of programs 
that cut across both agency and committee jurisdictions, uncovering 
waste, mismanagement, and abuses of authority. None of these types 
of oversight, it should be stressed, is mutually exclusive of the 
others. 

In performing these different types of oversight, Congress, its 
committees, and individual members have available a range of over- 
sight techniques to insure that (1) program objectives established 
by Congress are being met, (2) executive waste and dishonesty are 
eliminated, (3) agencies are properly administered, and, (A) execu- 
tive policies reflect the public interest. Some of these techniques 
include: "legislative" or "committee" vetoes, casework, hearings, 
and investigations, statutory controls, investigations by the Gen- 
eral Accounting Office, and the establishment of reporting require- 
ments for Federal agencies. 

While the range of oversight techniques is broad, there is rec- 
ognition by many legislators that Congress is deficient in the con- 
duct of oversight. During 1973 hearings before the House Select 
Committee on Committees, Member after Member testified as to the 
inadequacies of committee oversight. As one Representative said, 
"I have never felt that any committee was doing an adequate job on 
oversight, and I felt very strongly our committee was not doing a 
good job on oversight through the years." 33/ To strengthen that 



33/ Committee Organization in the House . Hearings before the House 
Select Committee on Committees, 93rd Congress, 1st session, p. 
310. 



46 



function, many oversight reforms have been adopted in recent years 
(see discussion infra) . Several possible factors explain the in- 
creased concern with oversight, and why Members have problems con- 
ducting systematic review of Federal programs and agencies. 

Factors Prompting Concern With Oversight 

The events associated with Watergate have spotlighted the im- 
portance of oversight. Some observers have specifically concluded 
that if congressional committees had been more diligent in over- 
seeing such agencies as the FBI, CIA, and IRS, many of the Watergate 
abuses might never have occurred. For example, the National Academy 
of Public Administration said in March 1974 that "continuing over- 
sight could have made Members of Congress aware much earlier of the 
centralization of power occurring within the Executive through the 
conversion of presidential advisers to 'assistant Presidents.'" The 
Academy recommended that Congress "give major attention to strength- 
ening its capacity to perform the oversight function, with particular 
emphasis on evaluating the performance of executive agencies." 34 / 
Oversight might also help rebuild public trust in the Federal Gov- 
ernment. 

Second, the 1973-1974 oil embargo focused congressional and pub- 
lic attention on the false assumption that the world capacity to pro- 
duce oil was in no danger and sufficient to support U.S. needs. Many 



34/ Watergate: Its Implications for Responsible Government . A re- 
port prepared by a Panel of the National Academy of Public Ad- 
ministration at the request of the Senate Select Committee on 
Presidential Campaign Activities, March 1974, pp. 71-72. 



47 



are now aware that the United States, at least in certain areas, may 
be moving from an economy of abundance to one of scarcity. Hence, 
legislators are more sensitive than ever to the need to eliminate 
waste and mismanagement in program administration, to make every dol- 
lar count, to authorize programs that will work, and to terminate or 
reduce the funding of those programs' which do not function properly. 

Third, particularly since the 1960s (the "Great Society" in- 
novations for example) the number of Federal programs has prolifer- 
ated. From the small businessman anxious to receive a loan from the 
Small Business Administration to the elderly person dependent on Med- 
icare payments, more and more citizens are affected by Federal pro- 
grams. And if things go wrong in those programs, then citizens will 
alert their Representative or Senator. Moreover, there are today per- 
haps more groups and associations, than in the past, which monitor the 
implementation of Federal programs. As an example, the Brookings In- 
stitution has recently published a study on Monitoring Revenue Sharing , 
which examines the general revenue sharing program, which is to be re- 
considered by the 94th Congress. 35 / 

Finally, there are numerous Representatives and Senators who are 
interested in strengthening Congress' oversight function. This is 
vital to increased oversight because even if there is a propitious 
oversight climate, legislators and outside groups must still capi- 
talize upon those political forces if oversight is to be strengthened. 



35 / Richard P. Nathan, Allen D. Manvel, Susannah E. Calkins and As- 
sociates, Monitoring Revenue Sharing (Washington, D.C.: The 
Brookings Institution, 1975). 



48 



Many Democrats and Republicans, conservatives and liberals, support 
improvements in oversight, as witness Democratic liberal John Culver 
(Iowa) and conservative Republican Dave Martin's (Neb.) strong sup- 
port for the creation of oversight subcommittees on all House commit- 
tees. 

Problems in Conducting Oversight 

Many people on Capitol Hill assume that systematic, comprehensive 
review of programs would be attainable if only Members were more dili- 
gent. But, as Professor Morris Ogul has stated: 36 / 

No amount of congressional dedication and energy, no conceivable 
increase in the size of committee staffs, and even no extraordinary 
boost in committee budgets will enable the Congress to carry out 
its oversight obligations in a comprehensive and systematic manner. 
The job is too large for any combination of members and staff to 
completely master. Congressmen who feel obligated to obey the let- 
ter of the law are doomed to feelings of inadequacy and frustration 
and are laid open to charges of neglect. Fortunately, at least for 
their own morale, the members of the Congress tend to focus on the 
immediate more than the impossible. 

Nevertheless, there are committees that lack sufficient staff to con- 
duct meaningful review work and (or) neglect to establish an oversight 
agenda determining which programs and agencies under their jurisdiction 
have review priority. 

Most of the time, oversight is performed selectively, even er- 
ratically, because the costs of conducting systematic and comprehen- 
sive review are generally perceived to be too high. Many Members sim- 
ply lack the time (or interest) to explore the numerous, and often 



36 / Morris S. Ogul. "Legislative Oversight of Bureaucracy," in Com - 
mittee Organization in the House , Panel discussions before the 
House Select Committee on Committees, 93rd Congress, 1st session, 
p. 702. 



49 



complicated, facets of programs. This is often tedious work with no 
guarantee that Members will reap any political reward, either within 
their own chamber or among their outside constituents. Moreover, many 
committees confront a heavy legislative work load of bills proposing 
new programs or expansion of old ones and must of necessity devote al- 
most full-time attention to the enactment of these measures. As a re- 
sult, little time is left for oversight. 

Some committee members have a narrow conception of when oversight 
should be done. As one chairman recently stated, "I am ready to do it 
(oversight) whenever there is evidence, but I want indictment first." 37 / 
Of course, one can argue that oversight is only necessary when an agen- 
cy's performance is found wanting and that committee-agency consensus 
on policy matters indicates little need for systematic review. It is 
not unusual, however, for committees to be "stacked" with members who 
share similar backgrounds and the values espoused by agencies. This 
encourages committee-agency consensus. Such consensus is a strong in- 
dication of the need for legislative review of that agency. 

Oversight can also be inhibited by the alliance that often de- 
velops between committees and agencies. Mutually supportive relation- 
ships can develop between the committees that authorize various pro- 
grams and the agencies that administer them. Of course, a committee 
may simply agree with a policy and see no need for oversight, or it 
may trust a Federal official so much that the committee sees no need 
to review the agency he heads. Moreover, as authors of legislative 
policy, committees may want program administration to "look good." 



37/ Committee Organization in the House. Hearings, p. 67. 



50 



The Legislative Reorganization Act of 1946 attempted to deal with 
these situations through the broad investigative authority granted the 
House and Senate Committees on Government Operations. Not tied to any 
single Federal agency or program, these committees can investigate pro- 
grams and agencies that are within the jurisdiction of another commit- 
tee. However, the Government Operations Committees have not been as 
effective as many Members and scholars had hoped. 

Among problems the Government Operations Committees have encoun- 
tered are synthesizing and transmitting information in a way useful 
for other committees and achieving some degree of coordination and 
cooperation in the sharing of information among the different types 
of committees. 

Another factor limiting oversight is the lack of clarity regard- 
ing legislative intent. Public laws are often the product of con- 
flicts and compromises, and when those compromises are translated into 
legislative language, it may not be possible to determine with any de- 
gree of assurance what specifically was intended for program adminis- 
tration. And on controversial bills, there may not even be any clear 
consensus regarding the goals or objectives of a program by committee 
members of staff. Hence, this sometimes leads to internal committee 
bickering about what constitutes clear standards for indices of pro- 
gram success or failure. Section 702 of the Congressional Budget and 
Impoundment Control Act (P.L, 93-344) recognizes this problem and di- 
rects the General Accounting Office to assist committees in developing 
statements of legislative objectives in order to facilitate program 
evaluation. 



51 



The way a committee functions can also determine the extent of 
oversight. Committees organized with rather independent subcommit- 
tees, adequately staffed and funded, are more likely to pursue over- 
sight than are committees where a strong chairman dominates his col- 
leagues. With established jurisdictions, subcommittees are better 
able to conduct realistic and manageable reviews of programs and 
agencies. 

Recent Oversight Reforms 

1. Legislative Reorganization Act of 1970 (P.L. 91-510) . The 
1970 act requires each standing committee to submit a biennial report 
on its legislative review activities; assigns new program review re- 
sponsibilities to the General Accounting Office; and directs the Con- 
gressional Research Service to provide greater substantive and policy 
analysis support to congressional committees. 

2. Congressional Budget and Impoundment Control Act (P.L. 93-344) . 
Title VII of the Budget Act is entitled "Program Review and Evaluation." 
The act authorizes committees to fulfill their oversight responsibilities 
by contract or by requiring Federal agencies to conduct evaluations. Com- 
mittees may also use evaluative techniques to pilot test programs or to 
do cost-benefit analyses. In addition, the General Accounting Office is 
directed to assist committees in developing statements of legislative 
objectives, and to establish an Office of Program Review and Evaluation 
to carry out the mission described in Title VII. 

3. Committee Reform Amendments of 1974 (H. Res. 988) . House 
Resolution 988, the Committee Reform Amendments, sought to strengthen 
the oversight role of the House Committee on Government Operations in 



52 



particular and that of other committees in general. The Government 
Operations Committee has been assigned the following new responsibili- 
ties: 

Preparation of an Oversight Report : Within 60 days after a new 
Congress convenes, the committee is to report to the House on the 
oversight plans of every committee and assist in coordinating the 
oversight activities of the House. The purpose of this proposal is 
to help facilitate coordination and cooperation among committees on 
oversight matters and to inform the House as to the oversight plans 
of the committees. 

Expanded Investigative Authority : H. Res. 988 made clear that 
the Committee on Government Operations need not refrain from investi- 
gating a subject area because another committee is studying, or has 
the jurisdiction to study, the same area. Before, the committee was 
often reluctant to investigate matters being considered by another 
committee. 

Summary of Review Findings to be Considered by Other Legislative 
Committees : Very often, the investigative findings of the Committee 
on Government Operations were not considered by the other standing 
committees during their legislative deliberations. H. Res. 988 makes 
clear that the pertinent review findings and recommendations of Gov- 
ernment Operations are to be considered by the authorizing committees, 
if presented to it in a timely fashion. Moreover, the report of the 
authorizing committee must include the findings and recommendations of 
Government Operations in a section of the report that is separately set 
out and clearly identified. 



53 



Oversight Findings Noted on Conmittee Reports : Committees are 
now required to indicate on the cover of their reports that they con- 
tain a summary of the oversight findings of the Committee on Govern- 
ment Operations when that is the case. 

All of the above changes were designed to strengthen the over- 
sight role of Government Operations and to reduce the isolation of 
committees. Very often authorizing committees are not aware of what 
Government Operations is doing and vice versa. Hence, neither prof- 
its from the work of the other. The changes are designed to elimi- 
nate that condition and to help promote cooperation and coordination 
among the committees. 

H. Res. 988 required other oversight. First, all committees now 
have the authority to review the impact of tax expenditures on matters 
that fall within their jurisdiction. Tax expenditures refer to money 
not collected because of various "loopholes" (tax subsidies, incentives, 
or preferences) in the tax code, which will total in fiscal year 1976 
nearly $92 billion. By reviewing tax expenditures within their juris- 
dictions, each committee will have a more complete picture of whether 
the financial assistance being made available to or through various 
programs is achieving national objectives. 

Second, each committee (except Appropriations and Budget) has a 
futures research and forecasting responsibility. On the theory that 
what was will influence what might be, then oversight and foresight 
are related. Committees now have the responsibility to identify and 
assess conditions or trends that might require future legislative ac- 
tion. Hence, the House may better be able to anticipate problems be- 
fore they become crises. 



54 



Third, the Select Committee developed the concept of "special over- 
sight," and H. lies. 988, as adopted, granted seven standing committees 
this responsibility. Issues are simply too complex to divide into water- 
tight compartments. Special oversight insures that certain committees 
have the right to conduct comprehensive oversight by enabling them to 
oversee programs and agencies that fall within the jurisdiction of an- 
other committee. 

Fourth, H. Res. 988 authorized committees to establish oversight 
subcommittees or to require their subcommittees, if any, to conduct 
oversight in their jurisdictional areas. At least nine House commit- 
tees have established oversight entities. 

Fifth, committee reports on measures shall include their over- 
sight findings separately set out and clearly identified. 

Finally, the costs of stenographic services and transcripts for 
oversight hearings will be paid from the House contingent fund rather 
than committee budgets. 

Conclusion 

Whether the surge of interest in oversight will continue or even 
accelerate is still unclear. For oversight is complex, difficult, and 
often a thankless process. Moreover, it may produce more questions than 
answers. Legislative evaluation of programs and agencies can perhaps 
identify what isn't working, but it probably won't answer the question 
of "what works?" While the problems that inhibit legislative oversight 
still exist, there is strong interest in this function, and this could 
mean that existing programs will be made to work better. 



55 



'PROGRAM EVALUATION CONCEPTS", GENEVIEVE J. KNEZO, 
CONGRESSIONAL RESEARCH SERVICE 



The objective of this brief is to discuss some of the basic con- 
cepts of program evaluation research, the other part of the theme of 
this seminar series. These remarks are intended to provide a frame- 
work to enhance the relevance of current evaluation research prac- 
tices to legislative oversight requirements. 

Federally funded program evaluation studies cost about $130 mil- 
lion in the fiscal year 1974. This is about a six-fold increase over 
the cost of such studies in the fiscal year 1969. Despite the pro- 
liferation of evaluation research, members of Congress, agency pro- 
gram administrators, and social scientists have expressed consider- 
able dissatisfaction with the quality and utility of such research. 
Generally it is acknowledged that many of the inadequacies of current 
evaluation practices come from a lack of understanding between the 
doers and the users of evaluation about the questions that evaluation 
can answer, and about which techniques are most appropriate to answer 
these questions. 

Two Types of Evaluation Research 

Federal agencies generally fund two types of evaluation studies. 
These are process evaluations and impact evaluations. They can be dif- 
ferentiated by their purposes, the techniques used to do them, their 
cost, and the time required to do the evaluations. 

Process evaluations generate information to answer management 
questions about the operating efficiency of a program. For instance: 
"who were the beneficiaries or target group? Were resources deployed 



71-867 O - 76 - 5 



56 



efficiently? What was the dollar cost per beneficiary? How many per- 
sons were required to administer the program? Generally was the pro- 
gram administered according to the management or delivery principles 
specified in the research design?" 

Impact evaluations answer questions about a more intransigent is- 
sue, that of cause and effect. For instance: "did the program inter- 
vention strategy or treatment effect a change in the participants? 
Was the change the one specified in the program?" 

Agencies use process evaluations to obtain information needed to 
make administrative changes in a program. The information elicited in 
impact evaluations is used to assist in making major policy decisions. 
For instance, "should a social innovation or new social program be im- 
plemented nationwide?" 

Process Evaluation 



Process and impact evaluations require the use of different re- 
search techniques. Process evaluations generally are non-experimental. 
They use data collected in the normal course of administering a pro- 
gram. Output data are indicated in terms of input data, for instance 
the dollar cost per beneficiary. These evaluations are inexpensive, 
require little time to do, can be done after the fact, or as a normal 
monitoring function. They can be done by program admins trators, either 
in the agency or in the field, or if extramurally, by public adminis- 
trators, auditors, economists, or social scientists. The Water Quality 
Improvement Act of 1970, P.L. 91-224, is an example of a legislatively 
mandated process evaluation. It requires the collection of data on 
the effectiveness of training programs. The act specifies as measures of 



57 



effectiveness, data on the number of persons trained and on occupa- 
tional categories for which training was provided. The Congress re- 
quested the information in support of making recommendations about 
developing new training programs. 

Impact Evaluation 

Impact evaluations typically use a quasi-experimental or an ex- 
perimental research design. Since one objective of the impact eval- 
uation is to determine if a program affected the participants' be- 
havior, impact evaluations must control for,or factor out, other 
events which might have caused modifications anyway in the partici- 
pants' behavior. This type of research requires the use of a control 
group and a test group. The members of each group must be matched 
on psychological, social, and economic characteristics relevant to 
the program. Since another objective of the impact evaluation is to 
determine whether the treatment should be adopted nationwide, the 
groups must be selected randomly. This means that in order for in- 
ferences or generalizations to be drawn about the efficacy of the 
program nationwide, all persons who ultimately might be given the 
program should have an equal chance of being selected as members 
of the experimental groups . 

The program or treatment, such as an income supplement or a job 
retraining program is given to the test group. The control group does 
not receive the program or is given a placebo, that is a different 
type of training program not being evaluated. Attitudinal and be- 
havioral changes in both groups are followed closely. Other factors 
which might affect the program outcome, such as an increase of 



58 



unemployment in the area, are noted and considered. Then an estimate 
is made of the actual impacts of the program on the test group. 

Impact evaluations are costly and require a long lead-time, usu- 
ally several years. They use the findings and techniques of social 
science research, such as hypothesis testing, survey research, and 
statistical manipulation of the data to draw and match samples, to 
collect and analyze data about the program's impact and about other 
trends or events which might affect the outcome of the program. Data 
collected in the normal course of administering the program are not 
sufficient for an impact evaluation since these evaluations may re- 
quire the collection of longitudinal data series, that is trend data 
overtime, on hard to measure attitudes and behavior. Appropriate 
data collection procedures usually have to be built into the program 
before it is implemented. 

Impact evaluations may be done on an anticipatory basis for pro- 
gram planning purposes. An example of a legislatively mandated impact 
evaluation is the Rehabilitation Act of 1973, P.L. 93-112. The Act 
requires the conduct of an impact evaluation, using test groups and 
control groups, to assess the impacts on participants of different 
types of rehabilitation training programs. One objective is to deter- 
mine the impact of the program on the participants' ability to adjust 
to disability and to improve their job skills. The New Jersey Nega- 
tive Income Tax Experiment is another example of an experimental im- 
pact evaluation. 

Reliable and valid information about a program's impact can be 
obtained only by the use of an experimentally based impact evaluation 



59 



strategy. However there are several drawbacks to experimental impact 
evaluations, in addition to those of cost and time required. These 
include: 

- a lack of basic social or behavioral science understanding of 
the causes and effects of human behavior; 

- problems of invasion of privacy due to the requirements of data 
collection procedures used in impact evaluation; 

- problems of equity of Federal spending practices if Federal 
subsidies are given to a test group and not to a control group or to 
the larger potentially recipient population; and 

- problems which result from diffuse or ambiguous statements of 
legislative program goals which prevent evaluation researchers from 
developing precise yardsticks against which to measure program ac- 
complishments. Such statements usually are vague because of the need 
for political compromise in writing legislation. 

In should be noted that quasi-experimental research designs, which 
do not always use randomly selected test and control groups, can some- 
times elicit useful policy relevant impact information. However this 
information generally is not as valid or reliable as that elicited from 
experimental designs. One type of quasi-experimental research design 
is a modification of the before and after case study. It compares pro- 
jected estimates of changes in participants' behavior, where predictions 
are based on previous trends or social science explanations about changes 
in human behavior, with observed changes after program is terminated. 
Like experimental designs, quasi-experimental designs also require pre- 
planning and a long lead time. 



60 



Concluding Observations 

These concepts of program evaluation have several implications 
for legislative oversight. First, both process and impact evalua- 
tions are useful for legislative oversight. However their purposes 
and limitations must be understood clearly. Process evaluations most 
appropriately are used to generate information about the administra- 
tive efficiency of on-going programs. Impact evaluations are used 
for anticipatory program planning and to assess whether a program 
intervention strategy should be adopted nationwide. Second, when 
reading evaluation reports, legislative staff should be aware of 
the purposes of the evaluation and the techniques used. This same 
caveat applies to the legislative staff person writing requirements 
for evaluation into legislation. Process evaluations cannot gen- 
erate information about program impacts. The most valid and re- 
liable impact evaluation information comes from evaluations which 
use randomly assigned control and test groups. Third, tradeoffs 
have to be made about the following factors when deciding whether 
to use an impact or a process evaluation strategy: cost, lead-time, 
the need to build requirements for data collection procedures into 
authorizing legislation; and the pros and cons of developing pre- 
cise statements of legislative goals. 

In conclusion, it should also be noted that evaluation is but 
one of many devices which can be used to generate information and 
analysis for oversight. Legislative adoption of even a pilot pro- 
gram reflects a political consensus about the desirability and feas- 
ibility of the program. No amount of evaluation, whether pro or con, 
will alter that consensus. 



61 



BIBLIOGRAPHY 



A. Background Material on the Use of Evaluation Information In 
Decisionmaking 



Knezo, Genevieve J. Program evaluation: emerging issues of possible 
legislative concern relating to the conduct and use of evaluation 
in the Congress and the executive branch. Multilith, Science Pol- 
icy Research Division, Congressional Research Service, Library of 
Congress, Nov. 16, 1974. 79 p. (75-35 SP.) (Included as Appen- 
dix.) 

Gilhert, John P., Richard J. Light and Frederick Mosteller. Assessing 
social innovations: an empirical base for policy. Forthcoming in 
Lumsdaine, A.R. and C.A. Bennett. Evaluation and experiment: some 
critical issues in assessing social program. Academic Press, 1975. 



B. Background Material on the Use of Evaluation Information in 
Decisionmaking and on Evaluation Methods 

Hatry, Harry P., Richard E. Winnie, and Donald M. Fisk. Practical 
program evaluation for state and local government officials. 
Washington, The Urban Institute, 1973, 134 p. 

Weiss, Carol H. Evaluation research: methods of assessing program 
effectiveness. Englewood Cliffs, Prentice-Hall, 1972, 160 p. 

Riecken, Henry and R.F. Boruch, eds . Social experimentation: a 
method for planning and evaluating social innovation. New 
York, Academic Press, 1974, 339 p. (H .61.R493, 1974.) 

Executive Office of the President. Office of Management and Budget. 
Evaluation and Program Implementation Division. Evaluation 
management: a background paper. Washington, Office of Man- 
agement and Budget, May 1975. 13 p. plus appendices. 

Abert, James E. and Murray Kamrass, eds. Social experiments and so- 
cial program evaluation: proceedings of the Washington Opera- 
tions Research Council Symposium. Cambridge, Mass. Ballinger 
Publishing Company, 1974, 199 p. 



C. Background Material Primarily on Methodology 

Bourch, Robert F. Bibliography: Illustrative randomized field ex- 
periments for program planning and evaluation. Mimeo, North- 
western University, Feb. 1974, revised, various pagings. 



62 



Cook, Thomas D. and Donald T. Campbell. The design and conduct of 
quasi-experiments and true experiments in field settings. May 
1974, Memeo, 165 p. To appear as Chap. 7 in M.D. Dunnette, eds. 
Handbook of industrial and organizational research. Chicago, 
Rand McNally, forthcoming 1975. 



63 



"ASK THE RIGHT QUESTIONS," DENNIS L. LITTLE, 
CONGRESSIONAL RESEARCH SERVICE 



In order to conduct a meaningful agency and program evaluation, we 
need to ask the right questions at the right time, and our inquiry must 
take place at several levels. The right questions can be asked only if 
the evaluator understands the agency's goals, missions, program, program 
objectives, and program and administrative evaluation criteria. In most 
cases such an understanding does not exist and, consequently, our evalu- 
ations are non-existent or of questionable utility. This analysis is in- 
tended to provide the reader with a framework for understanding and eval- 
uating the agency's reasons for existence (goals and missions) and their 
instruments (programs) for staying in existence. Hopefully, by using 
this framework, the reader can begin to formulate the appropriate evalu- 
ation questions. 

The goals of many Government agencies or departments frequently are 
non-existent, ill-defined, historically irrelevant, or hypothetically im- 
possible. The Commissioner of the Social Security Administration has 
been quoted as saying that he is not sure he knows what the goals of the 
Agency are, although he does know what they were in 1935. If the Com- 
missioner, Congressman Burke or the chief of staff of the Ways and Means 
Subcommittee on Social Security, and the Congressional Research Service's 
top Social Security expert were each to describe in one half -page the 
highest goal of the Agency, chances are good that there would be signif- 
icant variations in the three papers. The appropriate relationships among 
the goals, missions, and program components are represented by Figure 1. 
As illustrated in Figure 2, the highest goal of the Social Security 
Administration might turn out to be individual or family economic security. 



64 









EVALUATION I 


1 1 






i 

8 












1 § 

M 













E 




N s 




| N 




5 3 




5 £ 




if. 




P 3 




e 5 



65 







III 






V 


• • 




INDIVIDUAL OR 

FAMILY ECONOMIC 
SJ3CURITY 














o 













66 



This goal is composed of the program missions of income maintenance and 
reimbursement for services and commodities. The mission of income main- 
tenance must be subdivided further into operating concepts and eligibility 
factors, as Figure 3 emphasizes. 

These concepts are composed of insured rights, guaranteed rights, 
and needs-based rights. Insured rights include retirement, survivors and 
disability insurance programs. Guaranteed rights programs are not admin- 
istered by the Social Security Administration. The Supplemental Security 
Income program — adult categories of public assistance — is an SSA needs- 
based program. Entitlement and post entitlement are elements which deter- 
mine eligibility factors. These detailed components of the income main- 
tenance mission are pictured in Figure 4. 

This model of general structure and the specific example of the Social 
Security Administration provides a framework for proper interrogation at 
the program level by examining, at various sectors and intervals, the re- 
lated variables which affect the perceived goal. Prior to appropriation 
hearings the committee staff should reqvest from the chief administrative 
officer of the agency the following information: 

1) What are the five main objectives for each program? 

a) In an effort to reduce the information overload, one-line 
statements should be used to describe these objectives. 

b) A list of back-up, alternative and/or additional objectives 
may be furnished separately. 

2) What methods can be used to measure progress toward these objec- 
tives? 

a) One-line descriptive statements of the methods of measurement 
should be formulated. 



67 

















B§ 






O p 














Sa 










o 






II 











ae 






1 














w 






g 




■"■" ■ ' " 




g 




o 








o 



69 



b) The selected methods should not exceed five in number. 

c) The estimated cost of obtaining data on each indicator on hand 
if the data are not currently available and/or accessible. 

Upon being furnished this information, the committee should attempt 
an evaluation of the objectives and indicators provided. This might be 
done by the committee staff, the Congressional Research Service, the Gov- 
ernment Accounting Office, or some combination of the three. 

In conducting the analysis the following questions should be consid- 
ered: 

1) Do the objectives concur with the stated goals of the agency? 

2) Are these the most important objectives? 

3) Do the indicators measure program and administrative objectives? 

4) Are there better or more meaningful indicators? 

Upon completion of this analysis, the committee may ask the agency 
for additional information and a detailed analysis of the indicators. 
The committee's questions to the agency might include: 

1) What is the current level of the indicator? 

2) What is the desired level of the indicator? 

3) Over the next five years, what would be the cost — dollars, man- 
power, space, etc. — to close the gap between the actual level 
and the desired level of the indicator? 

An additional question not related to the specific problem is: What 
effect would attainment of this objective have on the attainment of other 
agency Adjectives? 

By structuring the investigator's perception of the agency, we can 
begin to structure our evaluation interrogation, and thus, hopefully, we 
can ask the right questions at the right time at the right level. 



DISCUSSION 

The issues raised after presentation of the first four papers 
dealt primarily with the meaning of the concept of oversight and 
methods which can be used to determine priorities for legislative 
oversight. Several points were made. First, frequently oversight 
is done by exception, meaning that many federally funded programs 
operate efficiently without congressional oversight; therefore the 
scarce resources available for legislative oversight should be used 
for handling only especially controversial or troublesome problems. 
Second, oversight can be considered a continuing process, whereby 
the agencies receive a continuous stream of management reporting 
information, and authorizing and appropriations committees receive 
annual budgetary information. This may result in information over- 
load. It was noted that perhaps attempts should be made to cut the 
flow of irrelevant information to agencies and committees. Third, 
political considerations frequently enter into the performance of 
oversight. However it is important for members of Congress to use 
their political barometers to identify important areas requiring 
oversight. 



71 



'CONGRESSIONAL RESEARCH SERVICE: RESOURCES FOR OVERSIGHT AND 

EVALUATION," NORMAN BECKMAN, DEPUTY DIRECTOR, 

CONGRESSIONAL RESEARCH SERVICE 



The mission of the Congressional Research Service for more than a half 
century has been to provide the Congress, and only the Congress, with research 
and reference assistance on the complete spectrum of public policy issues that 
Congress must consider. Recently, the Service was directed to establish as a 
goal massive aid for policy analysis in assisting the Congress to perform its 
legislative functions. 

In moving toward this goal of legislative support, it becomes apparent 
that there is no really significant distinction between providing policy analysis 
for pending legislation and providing analytical assistance for legislative 
oversight and evaluation. These classically discrete legislative functions are, 
in fact, part of a "push-pull" continuous process. Implementation of new 
legislation creates, after a period of time, the opportunity and need for over- 
sight. The evaluation process inherent in oversight activities uncovers the 
need for modification of existing legislation, termination of program authority 
or enactment of entirely new legislation. 

This is an important concept because the superficial tendency is to think, 
despite our checks and balances system, of Congress as being concerned solely 
with the law producing function and the Executive with the carrying out of 
national policy. It is easy to disregard the delivery side of policy before 
the legislation is enacted and to move on to other matters once the bill becomes 
a law. The ideal prescription for Congress is continuing review and oversight 



71-867 O - 76 - 6 



72 



but the temptation and frequent practice are to look for other legislative 
opportunities rather than to monitor how a program and policy is being managed. 

The theory behind the conduct of seminars on public policy issues organized 
by the Congressional Research Service for Congressional Members and staff is 
to collapse the time involved in the transfer of ideas. People with expertise 
and new knowledge to contribute are brought face to face with people directly 
involved in the very dynamic legislative process. The intervening and time 
consuming publication process for the transmittal of new information is by- 
passed through informal presentation and back and forth discussion and debate. 
In the case of this seminar series on Legislative Oversight and Program Eva- 
luation the experts are "us" - the Congressional Research Service, the General 
Accounting Office, the Office of Technology Assessment, and the congressional 
staff speakers. 

The basic authority for the Congressional Research Service in assisting 
the Congress to meet its oversight responsibilities is contained in the 
Legislative Reorganization Act of 1970. Several of the Act's provisions are 
directed to the Service's support of congressional oversight and evaluation 
activities: an explicit mandate for policy analysis of legislative proposals 
and oversight, new anticipatory functions-the submission of subject and termi- 
nating program lists at the beginning of each new Congress and a directive to 
maintain continuous liaison with all committees. The Service has no monopoly 
on this function-sharing it with a wide range of resources within the Congress 
(including Members' staffs, professional staff of committees, policy committees 
and informal research groups, the General Accounting Office, the Office of 
Technology Assessment, and the new Congressional Budget Office), and with such 



73 



outside resources as the executive departments and agencies, the various interest 
group organizations located in Washington and elsewhere, and universities and 
private research organizations. 

Following a very brief history and introduction to the Service, this paper 
speaks to four questions: What has been done to implement the specific new 
statutory mandates given the Service in 1970? What has been done to provide 
additional analytical support to the Congress for oversight? What are some of 
the new directions being considered by the Service? What are the implications 
for CRS of new proposals to improve Congress' research and information base? 
Following this paper, three speakers from the Congressional Research Service 
will present three different case studies of oversight staff support provided 
by CRS: 

(a) oversight activities stemming from CRS statutory responsibility 
to identify emerging subject areas of interest to committees and 
report on terminating program authorities; 

(b) oversight opportunities stemming from analysis of Executive 
Branch reporting to Congress in the field of foreign affairs; and 

(c) direct support for committees' oversight activities in the field 
of health care. 

Some other recently completed oversight projects conducted by CRS that 
come readily to mind (in order to preserve confidentiality of the request and 
response, only general titles are provided) include: mineral leasing; rede- 
ployment of U.S. armed forces in certain countries; implementation of revenue 
sharing; community development; education and recreation legislation; and 
retirement systems and immigration. 



74 






To emphasize the priority to be given by the Service to public policy 
research, Section 321 of the Legislative Reorganization Act of 1970 (Public 
Law 91-510) changed the institution's name from the Legislative Reference 
Service to the Congressional Research Service. The intent of this section 
of the act was to expand and change the nature of CRS support to congressional 
committees, emphasize the primary importance of assistance on legislative 
matters,- promote analytical and original research, grant the Service greater 
autonomy within the Library of Congress and render it more directly responsive 
to the Congress. 

What are the resources available "and how is CRS organized to provide 
oversight and evaluation support? The general intent of the 1970 act was not 
significantly different from that of the 1946 Legislative Reorganization Act, 
which emphasized committee support, appointment of senior specialists, and 
greater organizational independence for the Service. But this idealistic postwar 
act, like the Employment Act of the same year, never fully achieved its potential. 
In the first four years after the 1946 act, appropriations for the Service in- 
creased from $213,000 in 1945 to $514,700 in 1949 and its staff grew from 62 
persons to 117 in 1949. In contrast, budget increases during the first four 
years after the 1970 act raised the Service from a base of $5,653,000 and 363 
positions in FY 1971 to $13,345,000 and 703 positions in FY 1975. 

The Congressional Research Service is organized into ten divisions. Seven 
of the divisions, staffed by senior specialists, specialists, analysts and 
research assistants, perform substantive research in the following broad subject 
areas: American Law, Economics, Education and Public Welfare, Environmental 
Policy, Foreign Affairs (including national defense), Government, and Science 
Policy. The research divisions are supplemented by an interdisciplinary group 
of subject area specialists who comprise the Senior Specialists Division, a 



75 



Congressional Reference Division which handles the bulk of the ref- 
erence requests, and the Library Services Division which provides 
information and bibliographic support. 

The Joint Committee on the Organization of the Congress, in pro- 
posing the Reorganization Act of 1946 had "endorsed the principles 
that (a) research can be most effectively supplied by a pool of in- 
dependent experts, and (b) that the Congress should have direct ac- 
cess to its own separate research agency." In effect, Congress re- 
endorsed those less-than-fully achieved principles in the Legisla- 
tive Reorganization Act of 1970, with an added emphasis upon support 
for the functions of congressional committees. 

The House Committee on Rules, in its report on the Reorganiza- 
tion Act issued on June 17, 1970, emphasized the Service's new and 
expanded committee-related duties. "The complexity of committee 
responsibilities," the report said, "requires another supplementary 
staff to provide massive aid in policy analysis. For this purpose 
we propose that Congress expand the functions and facilities of the 
Legislative Reference Service. . . . Advance planning is essential if 
committees are to derive the maximum benefits from the resources to 
be made available by the Congressional Research Service." 

A number of organizational options were available to the Con- 
gress in achieving better oversight over the administration of laws 
and policy analysis assistance in examining new legislative propo- 
sals. The House Rules Committee rejected making additional staff 
available to each committee on the grounds that they would be ab- 
sorbed in day-to-day work, could not do specialized, indepth work, 
nor provide for exchange of information. 



76 



The Service is now the largest institution solely devoted to pro 
viding public policy research and information assistance to the Con- 
gress. It receives and responds to over 200,000 requests a year, 
which include over 400 assignments for indepth analytical assistance 
requiring a significant amount of manpower. CRS provides factual 
information, analysis of issues, alternatives to proposals, and eval- 
uation of alternatives without either advocacy or partisan bias. It 
also observes strict confidentiality of both requests and responses. 
Its assignments are received from four primary sources — committee 
staffs, members and their staffs, constituents (received through con- 
gressional offices) , and specific statutory requirements such as pro- 
vision of lists of terminating programs and activities and preparation 
of the Digest of General Public Bills and Resolutions. 

Implementation of New Statutory Responsibilities 

The legislative history of Section 321 of the Legislative Re- 
organization Act of 1970 indicates that the Congress sought to 
achieve two principal goals in revising the basic statute of the 
Service. First, it was clearly intended that the Service's research 
and analytical support for the committees of Congress should be ex- 
panded. Second, while sanctioning CRS assistance to committees and 
members in their representative functions and activities, it intended 
to make clear, as the House Report put it, that "the major function 
of the Congressional Research Service is to provide assistance to 
committees and individual Members of the Congress with respect to 
legislative matters." 






77 



The current operational goals of the Service are: 

- To provide indepth analytical and research assistance to the 
committees of the Congress to support them in their legisla- 
tive and oversight responsibilities; 

- To provide both members and committees with other analytical 
and research assistance to support their legislative respon- 
sibilities; 

- To assure that CRS assistance to the Congress is of the highest 
feasible quality and that it is responsive to the needs of the 
Congress ; 

- To provide information on the status and content of current 
legislative proposals; 

- To provide both information and reference assistance to both 
members and committees to support their legislative and over- 
sight responsibilities; 

- To assist members of Congress with their representative func- 
t ions ; 

- To provide the necessary administrative support to insure the 
optimal use of CRS resources in meeting congressional informa- 
tion and research needs. 

Stemming from these goals are four major new directives to the 
Service to be examined in more detail: (a) to provide policy analy- 
sis and research to assist committees in their legislative functions; 
(b) submission of lists of subjects and policy areas to committees 
at the opening of each new Congress; (c) submission of a list of pro- 
grams and activities scheduled to terminate to each appropriate com- 
mittee at the beginning of each Congress; and (d) maintenance of con- 
tinuous liaison with all committees. 

Policy Analysis for Committees . The statute (P.L. 91-510) di- 
rects that: "(a) Upon request, CRS is to advise and assist all com- 
mittees and joint committees of Congress in the analysis, appraisal, 
and evaluation of legislative proposals in order to assist them (1) in 



78 



determining the advisability of enacting such proposals, and of alter- 
natives to them, and, (2) in evaluating alternative methods for accom- 
plishing the goals of such proposals." 

This bold and idealistic directive to supply committees with ob- 
jective, indepth analysis and presentation of alternatives on any sub- 
ject matter raised a number of questions for CRS. Is it necessary to 
set up a separate research staff to provide these indepth services? 
What is the relationship between the jurisdictions of the congres- 
sional committees involved and the subject specializations of each of 
the existing CRS divisions? What gaps exist in subject coverage and 
competence within CRS? What kinds of methodology would constitute 
sound policy analysis research and what new analytical techniques and 
resources should be implemented by the Service? Can policy analysis 
leading to "determining the advisability of enacting each proposal" 
be provided without violating CRS policies of nonadvocacy? How rapid- 
ly can the Service expect to meet the large volume of unanticipated 
needs without compromising standards of quality and timeliness? How 
are priority research efforts to be determined? How can the possible 
fears on the part of professional committee staff about the impact of 
the Service's new duties be allayed? Should the Service concentrate 
on committees, already receiving considerable research support or on 
those committees receiving little or no assistance at the present time? 
Should contacts be made at the subcommittee level or channeled through 
the full committee? While the Service has explored each of these in 
depth, most will remain unanswerable until further experience has been 
gained. 



79 



As part of its evaluation and planning effort, the Service now 
identifies and monitors major committee projects that require analy- 
sis, have a direct connection to legislation, and require a signifi- 
cant investment of CRS manpower. During FY 1974, 456 such projects 
were initiated. 

A current analysis of CRS research for committees highlights a 
number of variables affecting the amount and type of work CRS provides 
congressional committees. The analysis notes that even with its new 
mandate, CRS progress is conditioned by factors which it has little 
or no control. The basic decisions that determine both the volume 
and the type of work, are made by the committees. 

Lists of Subjects and Policy Areas for Committees . One of the 
major new responsibilities assigned to the Congressional Research 
Service by the Legislative Reorganization Act of 1970 was designed 
to assist committees with their advance planning. CRS was thus di- 
rected to make "available to each committee of the Senate and House 
of Representatives and each joint committee of the two Houses, at 
the opening of a new Congress, a list of subjects and policy areas 
which the committee might profitably analyze in depth..." 

Here too a range of problems and options confronted the Service. 
Should a comprehensive list be prepared for each committee or should 
the lists be limited to subjects on which CRS could offer assistance? 
Should only lists be provided or should additional analytical materials 
be provided? Should lists be provided to committee chairmen only, all 
members of a committee, or to Congress as a whole? What methodological 



80 



procedures should be used in identifying and tracking policy areas in 
each major subject field? What if a committee does not desire the sub- 
mission of such a list? 

For the pilot implementation of this new responsibility, the fol- 
lowing procedures were developed and carried out. Late in 19 72, ap- 
propriate CRS staff met with committee staffs and, in some cases, mem- 
bers. Ad hoc teams consisting of some 30% (14A persons) of the total 
research staff were formed to work with the committees. Teams were 
composed of individuals knowledgeable in most aspects of a committee's 
jurisdiction and in many cases included staff from several divisions. 
The committee was encouraged to discuss its plans and needs for re- 
search with the appropriate team and in turn CRS staff suggested ad- 
ditional promising subject areas for the committee to consider. It 
was also an opportunity to describe CRS capability in the committee's 
fields of interest. The resulting lists of subjects and policy areas 
thus incorporated contributions by both CRS and the committee. 

Each committee was given the option of receiving a selected list 
consisting only of subjects on which the Service could provide assis- 
tance, or a comprehensive list, which included but was not limited to 
those subjects on the selected list. The final product, unless spe- 
cifically requested otherwise by a committee, included a summary list 
of agreed-upon subjects and policy areas, a one-page description of 
each subject to explain the reasons for its inclusion and provide ad- 
ditional sources of information, and, to the extent feasible and de- 
sired by the committee, additional materials to supplement and illus- 
trate the one-page treatment. 



81 



Terminating Programs . Documentation and current status of legis- 
lation has always been a concern of the Congress. The Reorganization 
Act inaugurated new documentation responsibilities by directing that 
CRS prepare for each committee a list of programs and activities, 
within its jurisdiction, scheduled to terminate during each Congress. 
CRS, in developing its guidelines, assumed that the principal purpose 
of preparing the lists was to further assist the committees in their 
advance planning efforts. 

Beginning in late 19 71, the American Law Division of the Service 
examined over 4,000 statutes enacted during the preceding decade and 
by December 1972 had identified some 730 programs and activities sched- 
uled to terminate during the Ninety-third and subsequent Congresses. 
Over half of these programs related to authorization of appropriations, 
and an additional 23% provided general authority for programs operated 
by executive branch agencies. Other categories included termination 
of commissions, loan authority, tax authority, reservations of land, 
pensions, and reporting requirements. 

Following identification of terminating programs, the Service pre- 
pared summary lists of the 458 programs and activities scheduled to ex- 
pire during the Ninety-third Congress and transmitted these, with sup- 
plementary descriptive materials, to 32 House, Senate, and joint com- 
mittees of Congress. Each committee packet included basic identifying 
information and a legislative history on each terminating program with- 
in the committee's jurisdiction. For 319 programs terminating during 
the first session of the Ninety-third Congress, the Service, insofar 
as resources permitted, also sent to each committee a more detailed re- 
port to aid the committee in reviewing the program. 



82 



CRS Liaison with Committees . The Legislative Reorganization Act 
requires CRS to maintain continuous liaison with all committees so 
that they may better understand and utilize the Service's multidis- 
ciplinary research resources and facilitate advance planning. Al- 
though separately stated in the 1970 act, several of the Service's 
new duties concerning congressional committees are closely related 
in overall purpose. Because preparation of lists of subjects and 
policy areas and terminating programs for committees was not re- 
quired until late 1972, the Service turned its attention to the gen- 
eral liaison function. 

The objective of the new liaison effort to be undertaken was de- 
scribed in a memorandum from the director (contained in the 1971 An- 
nual Report) stating in part: 

Our liaison efforts with committees should be directed toward: 
(a) keeping them informed about our present and potential re- 
sources for assisting them; (b) acquainting them with Service 
reports, already prepared or in preparation, of interest to 
them; (c) informing them about other, non-Service, materials 
of use to them and of techniques by which the Service can keep 
them regularly informed about such materials; (d) the prepara- 
tion and submission of those lists of subjects and policy 
areas the committees might profitably analyze in depth that 
we are obligated to supply them, including subjects of emer- 
ging interest to them; and (e) preparing and submitting the 
lists of expiring legislation also required by the Reorgani- 
zation Act. 

Perhaps the. major options facing the Service in implementing the 
liaison function were those of internal organization and rate of im- 
plementation. A recent internal memorandum states the basic organ- 
izational decision: 

One of the first questions the Service examined after passage of 
the 1970 Act was whether it should establish a staff exclusively 
devoted to committee liaison — that is personnel whose sole func- 
tion it would be to maintain communication and contact with com- 
mittees. After careful consideration, and at the urging of vir- 



83 



tually every senior CRS staff with experience in working closely 
with committees, the Service decided not to adopt this approach, 
at least during the first few years of operations under the 1970 
Act. Until circumstances indicated another course, therefore, 
we decided to rely upon our senior subject people, those who ac- 
tually do substantive work for committees, for the liaison func- 
tion. 

While a more formalized network might have been expected to be 

established by this time, the process to date has been an evolving 

one. This, in large part, is due to the number of committees and 

subcommittees involved, approximately 350. The goal continues to be 

an evolutionary approach designed to build working relationships based 

upon subject expertise. 

Other Innovative Services 

The Service's response to congressional needs goes beyond the 
very ambitious and explicit statutory provisions of the Legislative 
Reorganization Act. The House Rules Committee report on the Reorgan- 
ization Act directed CRS to experiment with additional ways of pro- 
viding assistance to Congress: 

Staffs should become proficient in existing analytical techniques 
and develop new ones as necessary. Both agencies [CRS and GAO] 
should be encouraged to experiment and to be innovative in carry- 
ing out their new as well as their older functions. 

The director of the Congressional Research Service, in recent 

testimony before the Joint Committee on Congressional Operations, 

identified certain new services not explicitly required by the 1970 

act: 

We sponsor several series of seminars at which outside experts 
discuss major public policy issues with Members or staff. We 
have steadily expanded our automation facilities and sought 
new and promising applications. One of these, for example, 
provides us, and therefore the Congress, with one of the largest 
and most up-to-date economic data bases currently available and 
includes programs for analyzing the economic impact of alternative 



84 



fiscal and monetary policies. With the encouragement of the 
Senate Committee on Rules and Administration, CRS is develop- 
ing an issues briefing system through which we will be able 
to provide concise updated information about current legisla- 
tive issues by video screens located in the offices of Members. 

The public policy seminars for members and staff, and the automated 

information services, are described in more detail below. 

Public Policy Seminars . The Service has expanded cooperative 
arrangements with outside organizations to permit members and their 
staffs to meet informally with experts on national issues. With 
the Advanced Study Program of the Brookings Institution, CRS ini- 
tiated in the fall of 1972 a series of seminars for members of 
Congress and a separate series for the senior staffs of member 
and committee offices. In the past two years, twelve seminars 
for members have included discussion of such topics as: U.S. re- 
lations with China, crime prevention and law enforcement, energy 
and the environment, national housing programs, consumer protec- 
tion, media and the government, and the multi-national corporation. 
Eight seminars have been held for congressional staff on: legis- 
lative implications of the 1972 election results, budget reform 
and control, legislation affecting the status of women, and new 
trends in strategic weapons policy in the national defense budget. 
The series has attracted an overall attendance of 277 members and 
677 staff. 

Additional seminar activities have included a continuing series 
on national urban growth policy, four seminars exploring basic tax 
issues, held for congressional staff in 1973, and presentations by 
CRS and the Office of Management and Budget in February 1974 to 
House, Senate, and CRS staff on use of the 1975 budget documents. 



85 



Automated Information Services . The Service has identified a 
number of procedures with potential for the storage and delivery of 
information through the application of data processing techniques. 
Thus, in recent years, the service has gained access to or has in- 
ternally developed a number of data files. The Legislative Infor- 
mation Data Base includes the contents of the Service's Bill Digest 
and additional information on all public bills and resolutions of 
the current Congress. Each bill is monitored for 22 data elements 
(for example, bill co-sponsor, committee action, identical bills) 
and information is retrievable via video tube in several formats 
(for example, a run of all bills assigned to a particular commit- 
tee and a listing of all bills relating to a specific subject). 

Most recently, at the initiative of the chairman of the Senate 
Committee on Rules and Administration, the Service has been devel- 
oping an automated Issues Briefing System. This experimental new 
service is being made available on a pilot basis, pending its fur- 
ther development and refinement. The system is designed to serve 
the current information needs of members, committees, and their 
staffs by maintaining general, easily accessible, summaries of many 
key issues of public policy. The summaries, or issue briefs, are 
provided by means of a computer information storage and retrieval 
system that can be accessed by video-tube terminal. Copies of in- 
dividual Issues Briefs are also available to both the Senate and 
House in printed form. Each Issue Brief (currently on 130 topics) 
contains current information on background facts, legislation, con- 
gressional hearings, reports, and chronological developments, and 
provides useful references. 



86 



In addition to these programs maintained by CRS personnel and 
stored in the library's computer, the Service currently has rental 
access to five outside data banks. The New York Times Information 
Bank is accessed by a videotube and associated high-speed printer 
and queries are transmitted by direct telephone line to the data 
files. These files contain indexes and abstracts of all news ar- 
ticles published in the New York Times since 1969 and abstracts 
from selected additional magazines and newspapers. MEDLINE con- 
tains over 400,000 entries from some 1,200 journals in the medical 
research and public health fields. Its bibliographic data is re- 
trievable by author or subject. JURIS (Justice Retrieval and In- 
quiry System) maintains the text of six information files, in- 
cluding the U.S. Code. Additionally, CRS has access to programs 
that provide the program grants and expenditures of the Depart- 
ment of Health, Education, and Welfare; that can project the costs 
of Federal support to education and other programs; and that pro- 
vide an econometric model of the national economy, which enables 
extensive analytical review of the Federal budget. 

The Past is Prologue 

In the past four years, CRS appropriations requests have placed 
the highest priority on implementation of new and expanded responsi- 
bilities under the 1970 act. Although the rate of growth has been 
slower than anticipated, the Service has acquired significant new 
staff resources, and assistance to committees has grown correspond- 
ingly. Now that the Service is moving toward completion of its five- 
year implementation program, new approaches are being explored to 
satisfy the research needs of the Congress. 



87 



A statement by the director of the Service at the May 1974 hear- 
ings of the Joint Committee on Congressional Operations on research 
and information support for the Congress sketches certain of these 
exploratory efforts: 

The Service has already asked for funds that will permit us to 
explore the addition of another dimension to our support for 
Congress: futures research. This relatively new discipline 
might help us to do a more penetrating job of identifying and 
analyzing emerging public policy issues.... 

We also hope to establish closer liaison with the scholarly com- 
munity, perhaps by a program of temporary assignment of scholars 
to the Service who could provide tangible assistance on specific 
projects and who would bring fresh insights to the staff of CRS 
and the Congress. 

Many Members and committees show increasing interest in the use 
of public opinion surveys to supplement their informational re- 
sources. Although it is probably inappropriate for CRS to con- 
duct such surveys, there are a number of ways in which we might 
assist Congress in this area. 

I have already mentioned our interest in establishing programs 
that would permit us to attach some of our staff to committees 
for limited periods so they can learn about the needs of those 
organizations . 

The actions recently taken by the House of Representatives, after 
consideration of the recommendations of the Select Committee on Commit- 
tees and substitute proposals, will also have a number of impacts on 
CRS, as well as on the delivery of research and information services 
to the Congress generally. Of major long term significance, there 
was created a study commission to make recommendations on the infor- 
mation problems of the House of Representatives. The study is to in- 
clude, but not be limited to, CRS, the General Accounting Office, and 
the Office of Technology Assessment and the organizational framework 
that makes them effective or ineffective. The House also created a 
legislative classification office and required the preparation of sum- 
nary descriptions of House bills. 



71-867 O - 76 - 7 



The enactment of the Congressional Budget Act of 1974 (P.L. 93-344) 
will also significantly affect the work of the CRS. As previously noted, 
the Service has already strengthened its capacity to perform certain 
kinds of budget analyses. It is anticipated that cooperative working 
relationships will be developed with the new Congressional Budget Of- 
fice, as directed in the statute, and with the budget committees. 

Other legislation introduced in the Ninety-third Congress to es- 
tablish new offices and commissions in the legislative branch would, 
if enacted, also impact on CRS and its plans for further implementation 
of the 1970 Legislative Reorganization Act. 

Implicit in certain of the proposals for improved congressional 
decision-making is a search for a "philosopher's stone," a device 
that will produce the right answers: a deus ex machina that will 
recommend legislation that best achieves the public interest. 

The Service, as a staff arm of a policy-mediated and partisan 
power structure, operates on an entirely different premise. It has 
no official views regarding which is a better solution to a legisla- 
tive problem. It attempts to identify choices and to forecast the 
apparent implications of the alternatives, but it is the member and 
then the Congress that make the decisions. 

Hairline distinctions must be observed by each CRS staff person 
on the one hand to avoid either advocacy or sterility and on the oth- 
er to maintain a reputation as a leading specialist in his subject 
field. The rules of the game are described in a recent Praeger vol- 
ume on U.S. government agencies, The Library of Co'ngress : 

In order for the CRS to survive, the congressional inquirer must 
never be able to guess where the researcher himself stands on any 
particular issue. The result is that if someone believes deeply 



89 



that a particular solution is the only answer, that person is 
inappropriate for the CRS.... But the person who believes deep- 
ly in the importance of his subject field and who is eager for 
the Congress to have as many solutions available to it as pos- 
sible and to have as great a knowledge of the impact of these 
alternatives, together with a sensitivity to what these solu- 
tions will cost this generation and five generations to come, 
and who yet recognizes that for the democratic system to sur- 
vive the choice of a solution must be left to the elected rep- 
resentatives — that is the person the CRS seeks for its staff. 

These and other proposals will continue to involve CRS and the 
Congress in carrying out the spirit of the Legislative Reorganization 
Act of 1970. The CRS mission remains to respond to the research, ana- 
lytical, and informational demands of Congress. To do so will mean 
constantly seeking and implementing new and improved ways of fulfil- 
ling that mission within the ground rules of confidentiality, non- 
advocacy, and nonpartisanship. 

References 

"Recent development of CRS research and analytical work for committees. 
In U.S. Congress. Joint Committee on Congressional Operations. 
Congressional Research Support and Information Services. A com- 
pendium of Materials. (Committee Print) Washington, U.S. Govt. 
Print. Off., 1974. (93rd Congress, 2d session.) pp. 38-63. 



90 



"THE SERVICE'S RESPONSIBILITIES FOR EMERGING ISSUES AND TERMINATING PROGRAMS' 
DAVID E. GUSHEE. CONGRESSIONAL RESEARCH SERVICE 



The Legislative Reorganization Act of 1970 charged CRS with two 
specific functions that relate at least partially to congressional 
oversight activities. These two CRS functions are: 

(1) To prepare and present to each committee at the beginning 
of each Congress a list of subjects and policy areas the committee 
might profitably pursue in depth. 

(2) To make available to each committee at the beginning of each 
Congress a list of programs and activities scheduled to expire during 
that Congress. 

Since enactment of the Legislative Reorganization Act, CRS has 
carried out these new statutory functions twice. What follows is a 
description of how the functions were carried out, what the list so 
generated contained, how the committees have responded, and what the 
exercises might mean to the oversight and program evaluation responsi- 
bilities of the committees. 

Basically, the CRS organizes around issues areas, some of which are 
discipline-based such as law or economics, while others are function- 
based, such as education, taxation, environment, or governmental structure. 
Either way, CRS analysts end up specializing in reasonably well defined 
issue areas which may or may not coincide with committee jurisdictions 
in the Congress. 

Some months prior to the end of a Congress, the Service sets up ad 
hoc task forces corresponding roughly with committee jurisdictions. These 
task forces review the information available to them about issues facing 
their committees in the forthcoming Congress, and develop a list of these, 
along with a brief analysis of the status of debate on each. 



91 



Input to these analyses includes a list of programs within that 
committee's jurisdiction which are scheduled to expire (how this is 
generated will be discussed later), the analyst's knowledge of the com- 
mittee's agenda, an awareness of debate in the current Congress on ini- 
tiatives not yet enacted into programs, an awareness of debate over 
agency performances in implementing previously enacted legislation, 
awareness of discussion in the professional and trade literature or 
"invisible college" in the issue areas or related areas and awareness 
of concerns and suggestions for improvement of the design of legislation, 
among other things. 

The ad hoc task forces thrash out, from this raw material, their 
view of issue areas their committees might either have to or want to 
deal with, along with some preliminary views on priorities and opportunities 
for useful research and analysis the Service might perform (or that others 
might perform, if appropriate). 

The task force coordinators, thus armed, initiate discussions with 
committee staff beginning at the staff director level and proceeding as 
far into the committee and subcommittee staffs as can be arranged. These 
have at least two purposes: 

(1) To reach agreement on a list of subjects and policy areas for 
formal transmittal from the Service to the committee. 

(2) To stimulate advance planning in the committees and to include 
in such planning such analytical and other support services the CRS might 
constructively contribute as extensions to the committee staff's capabilities 
of time, resources, special expertise, or objectivity. 



92 



In some of these discussions, other purposes are served beyond these. 
CRS subject specialists might deal directly with subcommittee or other 
staffers dealing with various issue areas, thus establishing a rapport 
from which future CRS support activities develop. In other cases, specific 
research projects are developed and agreed upon on the spot, thus giving 
the committee a planned flow of reports and giving the Service a chance 
to plan its own research agenda and resource development. 

In other cases, of course, the exercise results primarily in a 
formal transmittal only. Further work relationships are left for develop- 
ment by other methods. 

For the 94th Congress, the Service organized task forces for 49 
committees. The lists when transmitted all included some oversight topics, 
and a number of them included agreements on studies the CRS would either 
do or participate in as components of the committee's oversight activities. 

Subsequent to transmittal of the lists, of course, contacts between 
committees staffers and CRS specialists have led to many additional over- 
sight-related analyses by CRS for committees. We don't know for sure 
how many of these are direct or indirect results of the subject list 
exercise. 

I mentioned at the beginning that the CRS ad hoc task forces used 
as input to their own discussion the lists of programs scheduled to 
terminate during the next Congress. These lists are the result of a 
program initiated after the Reorganization Act of 1970, whereby each 
new Public Law is scanned to identify provisions with specified termi- 
nation dates. 

Each such program is entered on a computer file with a brief descrip- 
tion, a statutory citation, the expiration date, the committees of juris- 



93 



diction, and a brief legislative history. These entries are sorted by 
expiration date and by committee. 

For the 93rd Congress, there were 458 terminating programs, 319 of 
which were scheduled to expire in 1973. About 55% of the 458 were authori- 
zations of appropriations; the next largest categories were reports from 
agencies and commissions and "general authority." 

For the 94th Congress, 421 programs have been identified as expiring. 
The breakdown among categories is similar to that for the 93rd Congress. 
Most lists transmitted at the beginning of this Congress included issues 
associated with these terminating programs, and most terminating appropri- 
ations authorizations had issues associated with them that were controver- 
sial enough to be included, either explicitly or implicitly, in the issues 
treated. 



94 



"MONITORING REPORTS TO CONGRESS REQUIRED BY LAW", 
ELLEN C. COLLIER, CONGRESSIONAL RESEARCH SERVICE 



The requiring of a report from the executive branch is one of the 
oldest and most common methods by which Congress seeks to oversee the 
carrying out of the legislation it enacts. When a law contains a pro- 
vision such as that "The President shall report to Congress by January 
30 of each year on actions taken under this Act," Congress demonstrates 
its intention to follow through and see how the legislation is carried 
out. That it is an old and common method is shown by the fact that since 
1821 the rules of the House of Representatives have required that each 
year the Clerk present to each member a list of all reports which each ex- 
ecutive agency is required to» submit. The list submitted on January 14, 
1975 (House Document 94-7), was 44 pages long. But perhaps because re- 
porting requirements are so frequent, they are often not appreciated as 
the powerful tool for obtaining information that they are. 

There are many different kinds of reporting requirements. The most 
usual are for periodic reports, particularly an annual report. However, 
sometimes when Congress wants to keep a little closer watch on a program 
or agency, it calls for a semi annual or quarterly report. Another kind 
is for a non-recurring report as when an agency is called upon by Con- 
gress to make a single study and report its findings by a specific date. 
Still another kind is the requirement that Congress be notified of cer- 
tain Presidential determinations or findings, or of executive actions. 
For example, it is required that all international agreements not sub- 
mitted as treaties be transmitted to Congress within 60 days. 



95 



Reporting Requirements Study 

The effort to monitor reports required in the foreign affairs field 
began in 1969 when the Senate Foreign Relations Committee asked the For- 
eign Affairs Division for an identification and analysis of the report- 
ing requirements in legislation on foreign relations. The subsequent 
study, "Reporting Requirements in Legislation on Foreign Relations," 
published in 1970, described a patchwork of more than a hundred report- 
ing requirements spread through approximately 50 pieces of legislation. 

In a few instances, such as the Foreign Assistance Act, there were 
large numbers of reporting requirements, some of them overlapping or 
clearly obsolete. In other cases, there were no reporting requirements 
at all. For example, when Congress established the Central Intelligence 
Agency, it did not include any requirements for CIA to report to Congress. 
Nor were there any reports required from the Department of State on its 
operation or on overall U.S. foreign policy. 

Even in the first study it became clear that the reporting require- 
ments and the reports filed in response to them could be a valuable means 
of communication between the legislative and executive branches. For Con- 
gress, they could be an important tool for overseeing the carrying out 
of legislation. However, whether they were actually serving as such a 
tool required further study. There were two major questions, namely, were 
the reporting requirements being met, and was full use being made by Con- 
gress of the reports when they were submitted. 



96 



Study on Required Reports 

The Senate Foreign Relations and House Foreign Affairs Committees 
then joined together and asked the Foreign Affairs Division to make a 
survey of the extent to which the reporting requirements were being ful- 
filled by the executive branch. For the next two years, 1971 and 1972, 
the Division kept records of the date each report was submitted in re- 
sponse to a reporting requirement, and examined every such report to see 
if it fully met the reporting requirement and was generally useful for 
congressional purposes. 

The basic findings were published as a joint committee print in 
April 1973 in the study "Required Reports to Congress in the Foreign Af- 
fairs Field." The study found that the reporting requirements in legis- 
lation and the reports submitted in response to them constituted one of 
the major information systems used by Congress. A substantial amount of 
information was being furnished through the ever-growing number of re- 
ports. Nevertheless, despite the large number of reporting requirements 
and reports, the reporting requirement system was not being used to its 
fullest potential. It seemed clear that the effectiveness and efficiency 
of the reporting requirement system could be greatly enhanced. 

A number of common problems were found in the reports examined. The 
most obvious was lateness. A large number of reports were submitted too 
long after the -period they covered to be considered current, or too late 
to be useful in the hearings on a program. Many reports had security 
classifications and made no attempt to present a separate unclassified 
version, making it extremely difficult for any of the information in the 



reports to be utilized. Even some of the unclassified reports were dif- 
ficult to obtain. A surprisingly large number of reports were not dated, 
and many were not clearly identified as to which reporting requirement 
they fulfilled. 

On more fundamental matters, while some reports were excellent, many 
simply did not contain enough information to be useful to Congress. Often 
there appeared to be a real lack of understanding of the kind of informa- 
tion that Congress needed. 

The study found that required reports served a variety of functions, 
and that sometimes this led to a deviation from the main objective. For 
example, one function of requiring a report is often to assure the avail- 
ability of information to the public on a program. Sometimes, however, 
agencies submitted reports which probably were useful to those agencies 
for public relations purposes but which were of very little use to Con- 
gress in assessing the value and needs of the program. For example, some 
annual reports were attractive brochures but not the serious discussions 
of funds and problems needed by Congress. The problem did not always lie 
in the report. Often the reporting requirements themselves were vague 
and did not indicate when a report was wanted or what information was in- 
tended. 

The study indicated that although the executive branch prepared the 
reports and could do a lot on its own to improve them, there was also 
much that Congress needed to do. First, it needed to develop a system of 
monitoring the receipt of reports, so that there would be a central place 
for knowing whether a report had been received, and to whom it had been 



98 



sent. With such a monitoring system, if a required report was not submit- 
ted in a timely fashion, a question could be raised by the appropriate 
committee to the agency. Committees needed to gear the use of the reports 
into their preparations for hearings, so that the presentation material 
and questions at the hearings could build on the basic information in the 
reports. Evidence that the reports submitted were being used was needed 
to convince the preparers of the reports that their efforts were worth- 
while. 

Follow-up Study 

After the study on required reports was completed in April 1973, the 
chairmen of the House Foreign Affairs and Senate Foreign Relations Com- 
mittees sent it to the principal foreign affairs agencies and asked them 
for comments on the individual analyses of their reports. In addition, 
they asked the Foreign Affairs Division to continue to monitor the re- 
ports in the foreign affairs field and to assist in efforts to improve 
the reporting requirements. 

A joint committee print containing the replies of the agencies and 
discussing the improvement effort was then published in April 1974 under 
the title "Improving the Reporting Requirement System in the Foreign Af- 
fairs Field." That study pointed»out that 1973 had been a landmark year 
in the use of reporting requirements, inasmuch as the potential of such 
requirements had been applied to the fullest in the War Powers Resolution 
(P.L. 93-148), passed over the veto of the President on November 7, 1973, 
to place a check on the President's use of troops in hostilities in the 



99 



absence of a declaration of war. While the final sanction was a require- 
ment for congressional action if such hostilities were continued more 
than 60 days, reporting requirements were made the backbone of the plan 
to assure consultation with Congress. 

The study also showed two other needs. First, a long-range and con- 
tinuing effort was needed if reports were to be improved. Even when all 
agree on the objective, it takes time to define the information needed 
by Congress, additional time to change reports to meet new specifications, 
and still additional time after a report is issued to assess its useful- 
ness in the new form. Second, the reporting requirement system was an 
area in which cooperation between the executive and legislative branches 
could result in better serving the needs of both. 

Continued Monitoring 

The Foreign Affairs Division is now assisting the Senate Foreign 
Relations and House International Relations Committees in monitoring re- 
quired reports on a continuing basis. The basic tool with which the 
Foreign Affairs Division monitors reporting requirements is a chart con- 
taining a list of all the reporting requirements which are contained in 
legislation under the jurisdiction of the two committees or other legis- 
lation calling for reports which would provide information relating to 
foreign policy. The chart has six columns for listing (1) the legal ci- 
tation, (2) a brief digest of the requirement, (3) the agency by which 
the report is prepared, (4) the committee or receiver in Congress, 
(5) the date required and (6) the date any report is received under the 
requirement. A sample page of this chart is attached. 



100 



From the basic chart, which has some 300 requirements, a wall chart 
has been devised for keeping a record of approximately 60 requirements 
in legislation handled by the two committees which call for reports on a 
regular basis or by a specific date. This wall chart makes readily ap- 
parent whether reports are available or due, or whether they are late. 

In conclusion, there are two features of this project which are 
worth noting. First, the work has been done at the joint request of the 
Senate Foreign Relations and House International Relations Committees. 
This has permitted an economical use of the Foreign Affairs Division man- 
power, by serving two committees at once, as well as providing for a 
three-way exchange of ideas and information among the House committee, 
the Senate committee, and the Congressional Research Service. 

Second, the work has been sustained over a long period. The com- 
mittees have followed through on findings and recommendations and dem- 
onstrated their continued interest in required reports instead of drop- 
ping the subject after a brief exploration. 

The Foreign Affairs Division study has dealt only with reports in 
the foreign affairs field, where information is a particular problem 
for Congress. However, many of its findings would apply to reports in 
all fields. The oil crisis illustrated that foreign affairs is not the 
only field where lack of information is a serious problem. A study for 
the Commission of Government Procurement found very much the same situ- 
ation in regard to reporting requirements in the procurement field that 
existed in the foreign affairs field. 



101 



On every subject the requiring of report offers a way to generate 
and obtain needed information. Since the legislative requirement is the 
first element in this information system, it should be clear and well 
thought out. The second element is the report filed in response by the 
executive branch. Since it is the heart of this information system, it 
should be timely and fully responsive to the needs of Congress. The 
final element is the use of the report made by Congress. Since this is 
what determines whether the whole information system is worthwhile, it 
is incumbent on Congress to develop procedures to assure that the re- 
ports are fully used for the oversight and evaluation for which they 
were intended, or that the requirements for them are modified or deleted. 



102 



WO W rt 0) 



103 



„ " i 



tiomtimim 



.M?~.tm21Vte*** r: '**- r - msrr *T* r T-- 



.,.- ~c_- U , ISW 



104 



DISCUSSION 

One major point was raised in the discussion following the pre- 
ceding three papers. It was noted that generally the staff of the 
Congressional Research Service use their own independent resources 
and experienced judgments when developing emerging issues subject 
lists. However, it was acknowledged that some lists may represent 
a compromise and consensus of the issues raised by CRS suaff and 
staff of respective committees who will use the lists. 



105 



["THE GENERAL ACCOUNTING OFFICE'S RESPONSIBILITIES FOR OVERSIGHT AND EVALUATION 
IN REFERENCE TO THE CONGRESSIONAL BUDGET AND IMPOUNDMENT CONTROL ACT" ] , 
JAMES P. OLIVER, GENERAL ACCOUNTING OFFICE 



There are four phases in the Congressional Budget Process: The 
first begins when the President submits the current services budget by 
November 10. It will provide a budget based on a continuation of ex- 
isting programs at the same level of effort and will take into account 
economic factors. It is to provide a basis for the Joint Economic 
Committee's review of the current services budget and submission of the 
economic evaluation of the budget. It also provides a basis for the 
standing committees to begin analyzing the programs for which they are 
responsible. 

Next, the President submits the U.S. Budget for the ensuing fiscal 
year 15 days after Congress reconvenes. After the President's budget 
submission, all standing committees have a new responsibility. They 
have to submit their views on the budget to the appropriate budget com- 
mittee. All committees submit their views on the budget to budget com- 
mittees by March 15. 



106 



CO .H 

Pli M 
OS 

< 















z 






z 














c 






o 






z 














M ••^ 






O 








H 


LO 




os H m 


CO 




i— i 








3 


^ 




gS" 


K 




3 








CO 


>* 




CO >-" 


H 




.J 










t 




CO M <J 

H O S 


W 


H 


O 








u 




-J 


CO 


CO 








Uj 






co W 


g 


os 


w 








J 


W 




HJU 




OS 






En 




s 




P OS 


o 


fe 








O 


r o 


o 




oo o 


CJ 




H 


m 






z 


fe 




WZfe 


03 


z 


z 




CO 


o 




w 




BS H U 


<-t CO 


o 


eq 




CM 


z 


n; 


ca 


r~- 


M 03 


O CO 




2 


h 


o 


1— 1 






O 


W M 


CO W 


z 


s 


3 


-3- 


H 


os 


os 


\0 


uoioi 


os 


o 


p 




os 


c 


O 




zoo 


. O 




u 






O 


X 






< x 


u z 


H 


z 


►< 


r J 


Ph 


H 


z 


u 


> H z 


w o 


y 


o 


P5 


H 


w 


P 


O 


w 


S3S 


co o 


< 


o 


s -' 


co 


s 


< 


N -' 


CO 



CO M 

CO w § 

woo 

woo 

H P 

H«H 

M W 

S Z o 



W Ow. 

s2 



u W 

W hJ 

co <c 











CO 


H 




H 


CO 




H 


P W 


H 


OS 


W 






hJ M 


W 


o 


W 




£ 


O OS 


o 


PV 


H 




CO 


co P-, 


Q 


w 


H 




p 


w < 


5 


3 






CO 


PS 


03 


CO 


1 




w 


>* 

H ea 


hJ 


H 


O 




W 


z 


< 




o -^ 




H 


w w 


Z 


£ 


hH 




H 


OS CO 


o 


ca 






OS P 


W M 


3 


w 




1 


P o 


CM CO 


CO 


O W 




u pa 


O CO 




O M 


T3 


C 


z 


CM W 


w 


P OS 


H 


CJ 


o pa 


OS 


U 


pa p.. 


O 




o o 


• O 




< 


CO 


H 


HW* 


U Z 


W 


o 




W 


w o 


W 


H >- 




u> 


CO 


CO o 


o 


s_/ pa 


U 


p 


OS o 








w 


p 


M H 








CO 


pa 


W w 



w 
os 

OS o 

<c z 

co WO 

H OS 5h C_> 

M O 

2 W W OS 

oa < w 

P H U H 

W) W cfl h^ 

O W <3 CO 

HOW W 

Z P CO Z 

w ca o >-< w 
q z <: > 

M ^ M Q Z 

co oo p O 

W CO in U 

OS • Z i-H u 

P-, co W ^ PS 



f_l 


w 


s 


u 


pa 


M 


:=> 


> /-s 


CO 


PS O 




W rH 


to H 


CO 


uo Z 




o w 


H > 


vO o 


Z O 




w z 


• CO 


OS 


-o w 


PS >< 


W PS 


tD PQ 


CO pu 


o ^ 









H 
















W 








>- 








a 








ca 








a 




ft. 












s 




O 




CO 








CQ 








w 






s 






z 




W 






w 


U 




o 




H 








o 


w 






H 






> 




o 


H 










w 


> 




< 




1 




o 


CS 


OS 


z 


P 










w 


o 


J 




O 




S 




CO 




< 




u 




o 


CO 




CO 


> 








z 




H 


CO 


W 




H 


m 


o 


m 


z 








tJ 


m 


CJ 


w 


w 


S 


O 




O 


o 


w 


H 


DS 


sa 


M 




a 


^c 




H 


&s 


p 


s 


H 


p 




H 


M 


3 


CO 


o 


DJ 


ca 




z 




u 




z 


o 




a 


1— 1 








o 


p 


O 


u 


o 


o 


Px 




o 


p 


H 


CO 


•^ 


u 


o 


< 


w 


CO 


^-^ 





H 


W 


u 


■x. 


a 


H 


§ 


Z 


ca 








J 


CO 


< 


H 


z 


z 


O 


w 


M 


> 


CO 


w 


CO 




u 


os 


OS 


o 


r J 


-) 


z 


g 


O 

o 



107 



Committee Reports For The Budget Committees 

- The act requires that reports concerning the budget be prepared 
by every standing committee of both houses by March 15. 

- These reports are to be submitted to the committees on the budget 
of both houses and are to include each committee's views and 
estimates with respect to the content of the proposed concurrent 
resolution on the budget as it relates to matters within their 
jurisdiction. 

- Estimates are to be submitted of the total amounts of new budget 
authority, and budget outlays resulting thereform, to be provided 
or authorized in all bills and resolutions within the jurisdiction 
of the committee for the ensuing fiscal year. 

The report is particularly important for the authorizing committees 
since it represents their opportunity to justify the value of their pro- 
grams. We anticipate that they will want program evaluation information 
for this purpose. 

The Congressional Office submits a report to the budget committees 
by April 1. The report includes 5 year projections and a discussion of 
national priorities and alternatives. 

After getting the input from committees and the Congressional Budget 
[Office, each budget committee submits its proposed concurrent resolution 
by April 15. The scope of the concurrent resolution is important — Notice 
point (2) — This, in effect, is Congress's view on national priorities. 

First Concurrent Resolution 

(1) The appropriate level of total budget outlays and of total new budget 
authority; 

(2) An estimate of budget outlays and an appropriate level of new budget 
authority for each major functional category, for contingencies, and 
for undistributed intragovernmental transactions, based on allocations 
of the appropriate level of total budget outlays and of total new 
budget authority; 



KJs 



(3) The amount, if any, of the surplus or the deficit in the budget which 
is appropriate in light of economic conditions and all other relevant 
factors; 

Notice point (5) — This has been almost exclusively an executive 

initiative in the past. 

For Congress to develop a meaningful concurrent resolution, it is 
going to need good information, including program evaluations, and it 
will need it early in the session. 

The April 15 submission of the first resolution completes Phase 1 
of the budget process. Phase 2 starts with debate in both Houses under 
stringent time limitation leading to — 

First Concurrent Resolution Continued 



(4) The recommended level of federal revenues and the amount, if any, by 
which the aggregate level of federal revenues should be increased or 
decreased by bills and resolutions to be reported by the appropriate 
committees; 

(5) The appropriate level of the public debt, and the amount, if any, by 
which the statutory limit on the public debt should be increased or 
decreased by bills and resolutions to be reported by the appropriate 
committees; and 

(6) Such other matters relating to the budget as may be appropriate to 
carry out the purposes of this act. 

Completion of congressional action on the first concurrent resolu- 
tion by May 15. 

Also required by May 15 is the reporting of all authorizing legis- 
lation. 

In future years it will be easier for the authorizing committees 
to make this deadline since the act requires the executive to make ad- 
vance requests for authorizing legislation on or before May 15. For 
example, on May 15, 1975, the executive branch must submit its fiscal 
year 1977 authorization requests. 



109 



Phase 3 of the budget process begins May 16. It consists of review 
and enactment of spending bills. 

There is also additional input from the executive branch at two 
points in time. The President may transmit an amended budget on or before 
April 10 and July 15. 

Congress's review and enactment process continues through the summer 
until it completes action on new budget authority. This is required by 
the seventh day after Labor Day. 

Then, Congress completes action on the second concurrent resolution 
by September 15, directing committees concerned to take required action. 
Phase A between Sept. 16 and Sept. 25 consists of taking whatever action 
is required to complete action on a reconciliation bill by Sept. 25. 

The budget process is to be completed 5 days before the new fiscal 
year begins. 



110 







H hJ 






U J 






< M 






z « 


vC 


iTl 


W 


H 


Csl 


Z 

Q O 
Z M 


r- 


H 


< H 


Ph 


- 


< 


H 


H 


H M 


09 


CO 


Z J 
W M 

CO Z 
CO O 

u u 

CO W 
CO OS 















CO 














W _3 














H J 














U — 








os 






- PQ 








w 


H 




Pi 






^ 


H 


z 




X. z 






< 


W 


w 




O O 






^ 


•J 


^ 




U M 








E 


Oi 


10 


H 






OS 


^ 




CO < 






c 





•_ 




CO M 






^ 





z 




W kJ 






< 




c 


H 


OS t-i 


H 




•J 


■y. 


w 


h 


O CJ 


W 






■x. 




w 


z z 


CO O 




oS 


i- 


- 


v. 





W Q 




w 


OS 


z 




uu^ 


H 3 




£ 


a 


c 


> 


w [d u-i 


W PO 




Z 





= 


OS CM 


J 




<: 


O 


^ 






£g 












CJ Z • 

^ O H 




>• 






z 


o z 




< 


£i 


z 


c 


O P-, 


CJ 




a 


i-i 


c 




CO z u 


z 


>- 




O 




H 


O CO 


CO O 


H 


pa 


m 


z 


3 




CO 


— 


P 




c 


- 


CJ H P- 


w z 


2: 


2 






O 


w u pa 


OS O 


c 


p 


-J 


H 


CO 


CO < ^ 


O M 


PC 


> 


u: 





u: 




Z E- 


-- 


u: 


CO 


< OS 




o u 


p 


09 










o < 


< 


> -' 












H 
Z W 

M O 
Q 
CO 3 
H 03 
Z 

£3 

w z 
os2 

O CO 









Ill 



The Congressional Information Services Group in the Financial and 
General Management Studies Division has been organized to assist in pro- 
viding this information. The group has established five missions to be 
performed — 

For This Process To Work And Produce Better Budgets In Support Of National 
Needs, Information Must Be Provided To : 

. Congressional Budget Committees 
. Appropriations Committees 
. Other Committees 
. Congressional Budget Office 
. General Accounting Office 

There are staff activities addressing these missions. Some activi- 
ties participate in all missions; others do not as indicated. 



112 




XXX 

333 




333 

XXX 
XXX 

333 



333 
333 




X X X X 

3333 





2§ 



8:3 



O H Q 

►J U O 

W 3 33 

> os h 



i 

O B5 

Z O 

h h tn 

H Z Z 

« M O 

O M 

Oh U. H 

W O < 



z a 

85 



582 



CO C/5 

a w 

H to 

en w 



IB 



w o 

a: ^ 

35 



H 

z 

1 
> 

o 

OS Z 

£2 



H OS O 

M O M 
■ iJ t*« H 



Oh W < 

< w > 
CJ » u 



>2os 

Z W J 

m > s 5 
o o > 

o1°£ W 
O J o 

2S H z 

w o 

H 5 O H 

w w g 2 

0- H M O 

o ■< t* 

O H O Z 

O to H M 






113 



DISCUSSION 

A member of the audience asked Mr. Oliver if the GAO had already 
set up the OMB/GAO information system he described. He replied that 
the system is not yet operational because of understaf f ing. However 
the agency asked for new positions for this operation in its fiscal 
year 1976 budget. 



114 



'THE OFFICE OF TECHNOLOGY ASSESSMENT: A DESCRIPTION OF THE OFFICE'S 

OBJECTIVES AND ACTIVITIES:, GARY THOMAS, OFFICE OF 

TECHNOLOGY ASSESSMENT 



The Technology Assessment Act of 1974 (P.L. 92-484) authorized the 
establishment of an Office of Technology Assessment for the Congress as 
an aid to the identification and consideration of existing and probable im- 
pacts of technological application. 

In the legislation the Congress recognized that neither it nor the 
Federal agencies were designed to provide the Legislative Branch with ade- 
quate and timely information independently developed relating to the po- 
tential impact of technological application. The Office of Technology 
Assessment was therefore created to secure competent, unbiased information 
concerning the physical, biological, economic, social, and political ef- 
fects of the increasingly extensive and larger applications of technology'. 
This information is then to be used as one factor in the decision-making 
process in the Legislative Branch, particularly in those areas where Con- 
gress must reach judgments on technological applications and issues having 
technological implications. 

As created by the legislation, the Office of Technology Assessment (OT/ 
is an agency within and responsible to the Legislative Branch of the Govern- 
ment. Prior to the establishment of OTA, the General Accounting Office 
was the last legislative office created by the Congress; it was established 
in 1921. Since the establishment of OTA, a fourth legislative agency, the 
Congressional Budget Office, has been established. 

The basic function of OTA is to provide,, on a timely and independent 
basis, "early indications of the probable beneficial and adverse impacts 
of the applications of technology and to develop other coordinate informatic 



115 



which may assist the Congress." Eight specific activities are listed be- 
low which are identified in the legislation as the means of carrying out 
this function. The Office is required to: 

(1) Identify existing or probable impacts of technology or 
technological programs; 

(2) where possible, ascertain cause-and-eff ect relationships; 

(3) identify alternative technological methods of implement- 
ing specific programs; 

(4) identify alternative programs for achieving requisite goals; 

(5) make estimates and comparisons of the impacts of alternative 
methods and programs; 

(6) present findings of completed analyses to the appropriate 
legislative authorities; 

(7) identify areas where additional research or data collection 
is required to provide adequate support for the assessments 
and estimates described above; 

(8) undertake such additional associated activities as the ap- 
propriate authorities may direct. 

The OTA statute specifies that OTA shall consist of a non-partisan 
Congressional policy Board, an OTA Director and Deputy Director, a cit- 
izens Advisory Council, and such other employees and consultants as may 
be necessary in the conduct of OTA's work. The Congressional Board sets 
the policies of the Office and is the sole and exclusive oversight body 
governing OTA. The OTA Director is the chief executive officer and is 
responsible solely to the Board, of which he is a member. The function 



116 



of the Advisory Council is to advise on such technology assessment matters 
as may be requested by the Congressional Board. 

OTA's Congressional Board comprises six Senators and six Represen- 
tatives, evenly divided by party, who are appointed respectively by the 
President Pro Tempore of the Senate and the Speaker of the House. The 
current Board Chairman is Congressman Olin E. Teague of Texas and the 
Vice Chairman is Senator Clifford P. Case of New Jersey. The two posts 
rotate between the SEnate and House in alternate Congresses with the 
chairman chosen from the majority party, and the vice chairman chosen 
from the minority party. 

The Director of OTA is Emilio Q. Daddario, a former member of Con- 
gress who was instrumental in the development of the Technology Assess- 
ment Act. The Deputy Director is Daniel De Simone, a former White House 
science policy assistant. The Chairman of the citizens advisory council 
is Harold Brown, President of the California Institute of Technology. 
The Vice Charman is Edward Wenk, Jr., of the University of Washington. 

There have also been established for each of the assessment programs 
underway Advisory Committees of distinguished and knowledgeable experts 
regarding the specific issues being addressed by OTA for the Congress. 

* * * * 

When the Technology Assessment Board met in early 1974 to begin its 
work, having previously contacted each of the Congressional Committees, 
it found a number of priority issues before it. The following is a list 
of illustrative topics that emerged at that time: 



117 



Energy Policy — 

Impact of energy costs on food production and technological strategies. 

An assessment of the implications of existing projections of energy 
resources and requirements; feasible and forecasted technologies compared 
with these energy projections. 

Opportunities for conservation of energy; evaluation of the trade- 
offs among different kinds of conservation, and between conservation and 
expanded production capacity. 

Implications of large tankers, deep water ports, large storage capa- 
city, and related technologies. 

Alternatives to gasoline for automotive power plants: hydrogen, 
methane, methanol, and other possibilities. 

Off-shore drilling for gas and oil and alternative sources of hydro- 
carbon fuels: e.g., oil shale, tar sands, coal gasification. 

Growth Policy — 

Potential effects of modern technologies — e.g., agricultural, in- 
dustrial, sewage treatment, communications, law enforcement, transporta- 
tion — on population distribution and the growth of communities. 

Competing uses of land, including technical assessment of the im- 
plications of social values in conflict. 

Natural Resources — 

Technological opportunities for maximizing materials utilization: 
sustained growth in economic value of output with reduced consumption of 
materials. 



118 



Recycling — the implications of the closed cycle of materials usage. 

Substitution of materials in the context of resources/requirements 
forecasts. 

The management of all forms of materials information. 

Greater use of abundant materials in energy- limited industrial systems. 

The use of renewable resources for synthetic polymer; biomass as a 
new industrial raw material. 

The management of water: ground water, surface water, and induced 
rainfall; purification and desalting versus cascading of uses. 

The technological, economic, and political implications of the Earth 
Resources Satellite System: dissemination, management, and interpretation 
of data therefrom. 

The technological aspects of foreign policy planning for the 3rd Law 
of the Sea Conference. 

Hydrocarbons as alternative sources of protein for human and stock 
food. 

Urban Development and Transportation — 

Technologies for rapid, large-scale tunneling and excavation: their 
potential application and social effects. 

Implications for CATV and the "wired city" concept. 

Mobile home technology and its effects on public health, safety, util- 
ity services, education, security, local tax revenues, social and munici- 
pal costs (an example of "low" technology development with wide-ranging 
consequences). 



119 



Implications of possible shifts to various forms of mass public 
transportation, intra-urban and inter-urban 

Municipal sewerage: disposal versus utilization. The related prob- 
lem of animal waste management, disposal, and utilization. 

Noise control in high density areas; trucks and airports, industrial, 
heavy machinery, etc. 

Health Policy — 

The increasing cost of health care and delivery; opportunities for 
technological innovation in health care delivery systems to reduce costs 
and improve services. 

Adequacy, limitation, and utilization of scientific advice in the 
establishment of safety, environmental, and health standards. 

Trade Policy — 

Effect of international standards, measurement practices, and certi- 
fication procedures on U. S. trade. 

Technology as the basis for U. S. competitiveness in international 
trade. 

Trade implications of U. S. support for the development of regional 
technological systems (e.g., Mekong, Rio Bermejo, Northeast Brazil). 

Because OTA was just beginning its work and had limited means at its 
disposal, it had to be discriminating in the selection of its activities. 
To accomplish this, the following criteria were used: 

o Can OTA make a unique contribution, or is it 
within the normal capabilities of a committee 
of the Congress to perform this assessment? 



120 



o What is the magnitude of the alleged or 

expected costs and benefits to society of the 
various technological options involved and how 
are these costs and benefits distributed? 
o Is the technological impact irreversible? 
o How imminent is this impact? 
o Do we know enough to perform the assessment? 
o Is it of manageable scope? 
o What would the assessment cost? 
o How long would it take? 
o What is the likelihood of Congressional 
action? 
The attached matrix shows most of the requests we have received and 
the actions taken thus far. An assessment of drug bioequivalence has 
been completed and other on food, solar energy, urban mass transportation, 
the extraction of oil and gas off the coastal United States, industrial 
materials, super-tankers and super-ports, health, and international trade 
are all in various stages of development. 

To illustrate the process of technology assessment and how the as- 
sessment will assist the Congress in resolving issues with a technical 
content, let us consider one of the OTA assessments programs in more de- 
tail. The specific current assessment programs include energy, food, 
health, materials, oceans, transportation, technology and world trade, 
and exploratory assessments. The materials program has several assess- 
ments in progress simultaneously each of which is responsive to specific 



121 



Congressional requests. The materials assessments are related, however, 
and have been designed to collectively develop an overall materials pro- 
gram. We shall therefore choose the materials program for detailed dis- 
cussion. 

The OTA materials program was initiated by a study of major issues 
in national policy, as already identified by major study groups of na- 
tional stature, and of major legislative interests in materials. The 
materials science and engineering community has long recognized that the 
cycle of materials usage, including the associated use of energy and en- 
vironmental impacts, is central to civilization. Let us briefly consider 
this cycle because it is a fundamental concept to the OTA materials pro- 
gram. Materials flow through successive stages of production, use and 
disposal, creating an interrelatedness of raw materials, bulk materials, 
engineering materials, products, and waste. Some feeling for the size 
of materials flow in the United States is given by recognizing that the 
use per person per year is a total of 42,500 lbs. composed of 20,550 lbs. 
of nonmetallic materials, 17,300 lbs. of mineral fuels, 1,340 lbs. of 
metals and 2,310 lbs. of organics. 

To supply this cycle in 1973 the United States imported $19 billion 
worth of minerals and exported $11 billion worth. The nations of the 
world are inextricably tied together by the flow of materials. In the 
words of Edmundo de Alba, they are "condemned to interdependence". Con- 
servation of materials through more effective utilization and other means, 
such as recycling, will probably become increasingly urgent but will have 
far-reaching direct and indirect consequences on the economy, employment, 
etc. 



122 



Twenty two years ago the Paley report drew attention to the enor- 
mously increasing use of mineral raw materials, to the problem of meeting 
demand in the future, and to the need for conservation of materials. The 
world-wide boom of the 60 's and 70' s, fueled by inexpensive petroleum, 
obscured this underlying need. Discussion of the need for conservation 
and effective utilization of materials did continue but without the ur- 
gency of immediate shortages. Urgency began to develop in the early 70' s, 
however, as shortages began to occur and rose to a high level with the 
occurrency of the OPEC petroleum embargo and the threat of similar ac- 
tions for other materials. Thus, by mid-1973 there was widespread rec- 
ognition that the United States had a general, serious and long-term 
problem with supplies of essential fuel materials and other raw materials. 
This problem was felt most acutely in the case of the absolute, though 
temporary, shortage of oil, but it was felt also in the form of higher 
prices for other raw materials. Considering the total materials cycle 
again we would expect shortage in fuel or other materials at the begin- 
ning of the cycle to have consequences throughout the cycle as has indeed 
happened. These intermediate stage shortages are not entirely caused by 
raw materials shortages but also by limited productive capacity which in 
turn is tied to economic gactors. The entire system is under pressure 
from the rising cost of energy and theneed to reduce pollution. Still 
another pressure on the system is increasingly felt at the disposal end 
because availability of land for waste disposal is decreasing and the 
cost of disposal is increasing. Indeed, this is one of the principal 
driving forces for increased recycling. The question of reconciling the 



123 



need for accessibility to minerals with other uses of land and for pre- 
servation of the environment puts still another pressure on the system. 

In developing a program of technology assessments on materials, the 
Office of Technology Assessment has proceeded in stages. The first step 
was the development of a prospectus for materials assessments based on 
the study mentioned above. This prospectus was prepared with the assis- 
tance of the Congressional Research Service. The second step was to have 
the Materials prospectus reviewed by the Technology Assessment Advisory 
Council and by the Materials Advisory Committee. The committee members 
are: 



Earl H. Beistline 
University of Alaska 

Seymour L. Blum 
Mitre Corporation 

James Boyd 
Materials Associates 



Harry H. Herman, Jr. 
Consulting Engineer 

James A. Kent 

Michigan Technological Univ. 

Hans H. Lansberg 
Resources for the Future 



Lloyd M. Cooke 
Union Carbide Corp. 

Frank Fernbach 

United Steelworkers of America 



Elburt F. Osborn 

Carnegie Institution of Washington 

R. Talbot Page 
Resources for the Future 



Edwin A. Gee 

E. I. De Pont de Nemours and Co., Inc. 



Nathan E. Promisel 
Consultant 



Bruce Hannay 

Bell Telephone Laboratories 

Bruce Hannon 
University of Illinois 

William J. Harris, Jr. 
Association of American Railroads 



Lois Sharpe 

League of Woman Voters 

George A. Watson 
Ferroalloys Association 

Jack H. Westbrook 
General Electric Company 



Julius Harwood 
Ford Motor Company 



124 



Also, a conference on National Materials Policy was organized by 
the Federation of Materials Societies to deal with some of the issues 
raised in the Materials prospectus. This conference was held at Hen- 
niker, New Hampshire in August 1974. A series of workshops reported 
their recommendations including: 

- Improved management of existing materials information systems; 

- The need for a national stockpile of materials; 

- Economic and institutional barriers to innovation should be 
removed; 

- Motivation should be provided to encourage materials conserva- 
tion in the design stage; 

- Barriers to materials recycling should be overcome by strong 
economic incentives and effective institutional arrangements; 

- Some materials goals are shared by all countries and should 
provide the basis for concerted international program; 

- There was a need for a permanent national materials commission; 

- Consideration should be given to organizing a World Materials 
Year. 

The third step was to have a Materials Committee adopt a set of 
goals for the overall strategy of the materials assessments program and 
develop a recommended list of individual assessments. The Committee de- 
cided to accept the goals of the National Commission on Materials Pol- 
icy as a broad strategy. These goals are: 

(1) Provide adequate energy and materials supplies to satisfy 
not only the basic needs of nutrition, shelter and health, 
but a dynamic economy without indulgence in waste. The 



125 



other general goals of the NCMP support this primary goal 
and suggest a framework of policy constraints. 

(2) Rely on market forces as a prime determinant of the mix of 
imports and domestic production in the field of materials 
but at the same time decrease and prevent wherever necessary 
a dangerous and costly dependence on imports. 

(3) Accomplish the foregoing objectives while protecting or 
enhancing the environment in which we live. 

(4) Conserve our material resources and environment by treating 
waste materials as resources and returning them either to 
use or, in a harmless condition, to the ecosystems. 

(5) Institute coordinated resource policy planning which recog- 
nizes the interrelationships among materials, energy, and 
the environment. 

With these goals in mind and with the entire materials cycle under 
consideration, the materials advisory committee recommended the follow- 
ing list of assessments on September 20, 1974: 

(1) Devise materials information systems for reliable input to 
policymaking. 

(2) Establish a national stockpile policy. 

(3) Stimulate efforts to hasten materials recycling. 

(4) Develop means to encourage and assist manufacturing industries 
to us materials in fabricating products employing materials 
more effectively. 



126 



(5) Manage materials so as to conserve energy, but in a manner 
to minimize economic and social dislocation. 

(6) Expand and strengthen domestic minerals industry. 

(7) Stimulate education, research and development in the mineral 
extraction and processing fields. 

(8) Assess the interaction of environmental concerns with effec- 
tive utilization and production of materials. 

(9) Manage materials so as to conserve materials, but in a man- 
ner to minimize economic and social dislocation. 

(10) Examine land use in relation to laws regarding mineral ex- 
ploration and production. 
Assessments on materials information systems and on national stock- 
pile policy are now underway. Assessments are also being designed in 
the areas of: a) resource recovery, materials recycling and reuse, and 

b) institutional constraints on domestic mineral accessibility and c) 

c) conservation of materials. Further assessments on other aspects of 
the cycle of materials use are under consideration. 

These assessments were developed in response to Congressional re- 
quests. These requests included not only the first set received early 
in 1974, but also a later set which were received after the Materials 
Advisory Committee's recommendations and dreq upon these recommendations. 
These latter were more specific requests within the broad areas described 
in the first cycle of requests for materials assessments. 

The House Science and Astronautics Committee, which has since be- 
come the House Science and Technology Committee, made a broad request. 
In their first letter, dated January 22, 1974, Chairman Teague and 



127 



ranking minority member Mosher emphasized four areas for possible OTA 
study, including a technological data base for Congress and research and 
development programs to lessen United States dependence on importation 
of critical materials. In their second letter, dated December 13, 1974, 
they specifically requested studies of 1) materials information systems, 
2) national stockpile options, and 3) reuse of materials. 

The Senate Commerce Committee, in a letter from Senator Magnuson 
dated January 24, 1974, requested studies of the solid waste problem 
including 1) reduction of waste at the source, 2) recycling and resource 
recovery, and 3) energy recpvery. 

Representative Morris Udall, a member of the Technology Assessment 
Board, in a letter dated September 19, 1974, raised several questions 
to be answered by assessments: 

"What means do we have to deal with impending resource 
scarcities? What kind of roles will such methods as 
substitution of new materials for scarce ones, rationing, 
altered pricing systems, reuse and recycling, new effi- 
cient production technologies and new regulations gov- 
erning land use play under these conditions?" 
Senator Ted Stevens, a member of the Technology Assessment Board, 
requested a study of mineral accessibility in a letter dated November 6, 
1974. 

In addition to the interest in materials shown by these specific 
requests to OTA, there is widespread Congressional interest in materials. 



128 



As Chairman of the Senate Commerce Committee, Senator Warren 
Magnuson in a letter dated January 15, 1975, endorsed a request by 
Senator John Tunney for a study of means of conserving materials 
through reducing wastage of materials by reducing corrosion and other 
wastage processes. Senator Magnuson asked for: "1) an assessment of 
the kinds and amounts of materials wastage; 2) techniques for reducing 
wastage; and 3) technical and institutional impediments to applying 
these techniques." 

Over 140 separate bills on Materials were introduced into the 93rd 
Congress in the areas of Materials management, Materials recycling, 
Materials shortages, and Materials stockpiling. One bill which sub- 
sequently became law established the National Commission on Supplies 
and Shortages. 

The Materials Information System and National Stockpile Assessment 

are pertinent to this Commission which was established by Public Law 

93-426 on September 30, 1974. The Commission was required to make 

recommendations, no later than March 1, 1975, 

"with respect to institutional adjustments, including the 
advisability of establishing an independent agency, to provide 
for a comprehensive data and storage system, to aid in examination 
and analysis of the supplies and shortages in the economy of the 
United States and the rest of the world." 

Until June 30, 1975, the Commission may make such other reports and 

recommendations as it thinks appropriate. The Congressional findings 



129 



which led to the establishment of the Commission were as follows: 

1. The United States is increasingly dependent on the Importation 
from foreign nations of certain natural resources vital to 
commerce and the national defense. 

2. Nations that export s-jch resources can alone or in association 
with other notions arbitrarily raise, the prices of such 
resources to levels which are unreasonable and disruptive 

of domestic and foreign economics. 

3. Shortages of resources and commodities are becoming increas- 
ingly frequent in the United States, and such shortages cause 
undue inconvenience and expense to con umers and a burden 

on interstate conferee and the Nation J cono'my. 

4. Existing institutions do not adequately dentify and ar'ici- 
pate such shortages and do not adequately monitor, study, and 
analyze other market adversities involving specific industries 
and specific sectors of the economy. 

5. Data with respect to such shortages and adversities is collected 
in various agencies of the Government for various purposes, but 
is not systematically coordinated and disseminated to the 
appropriate, agencies and to the Congress. 38/ t 

"The dates have apparently been changed to June 30 and December 31, respectively. 

The OTA accordingly proposes to, assess pertinent features of 

materials information systems in terms of their past, present, and 

expected stages of development. Major deficiencies in the existing 

information systems should be identified and alternatives for their 



38 /(See section 720 of the Defense Production Act of 1950 as amended, 
50 U.S. C. App. 2169. The portions quoted above are subsections (h) 
and (b) attachment 1). 



130 



removal described and evaluated. The establishment of a materials 
information system may call for the creation of a new comprehensive 
information system, expansion of the present activities, or the establish- 
ment of an institution charged with insuring efficient and effective 
use of existing systems. 

The law establishing the National Commission on Supplies and 
Shortages also charges the Conmi.ssion to report 

"necessary legislative and administrative actions to develop 
a comprehensive strategic and economic stockpiling and 
inventories policy which facilitates the availability of 
essential resources" (sec. 720 (q)(5) of the Defense 
Production Act of 1^50 as amended; 50 USG App. 2169. Under- 
lining supplied). 

Accordingly, the OTA proposes to examine the consequences of alternative 

national stockpile policies. The possible uses of a national stockpile 

for broader public purposes, in addition to the limited national security 

purposes for which the "Strategic and Critical Materials Stockpile" was 

established, will be assessad. 

/ The problem is to identify and define the purposes that could be 

! 

served by a Government-managed or Government-financed institution that 
bought, held, perhaps upgraded, and sold selected materials; to determine 
the criteria governing the management and operation of such an institution ; 
to explore and report on the direct and indirect consequences of such 



131 



operation; to identify ways of enhancing favorable consequences and 
ways of mitigating or avoiding unfavorable consequences; and to assess 
options and alternatives in the design and operation of various institu- 
tional arrangements for performing the function in question. 

The second assessment, on national stockpile policy, will examine 
the attributes and consequences of alternative national stockpile 
policies. The possible uses of a national stockpile for broader pur- 
poses than the limited national security purposes for which the "Stra- 
tegic and Critical Materials Stockpile" was established, will be 
assessed. 

This assessment started in March, 1975; the final report is due 
in September, 1975. 

The third assessment, on resource recovery, materials recycling 
and reuse, will examine the institutional barriers to, and incentive 
for, achieving substantial resource recovery from urban refuse using 
the best current technology. To accomplish this task, assessments 
of interrelationships among (1) technology requirements, (2) economics, 
(3) institutional barriers and incentives, and (4) social factors will 
be made. The assessment will include consideration of barriers to 
policy-making and to decision-making as well as to operational imple- 
mentation of resource recovery technology. 

The fourth assessment, on institutional constraints on domestic 



132 



mineral accessibility, will consider the potential effects of modifying 
the structure of Federal laws and other institutional factors affecting 
the accessibility of domestic mineral resources. This study will 
include consideration of all steps leading to and including the ap- 
plication of technology for the purposes of mineral exploration, 
development, extraction, processing, and delivery. 

The fifth proposed assessment, on Conservation of Materials through 
reduced wastage, will examine types of materials wastage such as cor- 
rosion and wear. Estimates of the magnitude of loss will be made and 
estimates of the savings which could be accomplished through improved 
design and through application of protective procedures in use. The 
impacts of reduced wastage will be assessed including estimated 
impacts on the economy and on employment. 



133 



ASSESSMENT REQUESTS BY COMMITTEE 

FOR THE PERIOD DKCEMBE* 1973 THROUGH MARCH 19 75 



H ouse Committee on Agriculture 

2/13/75 Chairman Foley requests assessment of feasibility of using 
rice-blended food products as material eligible for export 
under Title II of the P.L. 480 program. 

House Committee on Appro p riations 

* 11/19/74 Chairman Mahon, on behalf of Chairman McFall and Ranking 

Minority Member Conte of the Subcommittee on Transportation 
Appropriations, requests a technology assessment with regard 
to automobile crash recorders. 

House Committee on Foreign Affairs 

1/18/74 Chairman Morgan suggests assessments concerning: 

1) Arms control 

2) Food 

3) Technology transfer 

4) Population/family planning 

House Committee on J udiciary 

**12/19/7"3 Chairman Rodino requests assessment of: 

** 1) Existing offshore oil and gas operations 
** 2) Effects of tripling present acreage of leases for oil 
and gas production on Outer Continental Shelf 
3) Detailed assessment of oil and gas operations on 
Outer Continental Shelf 

House Committee on Merchant Marine and Fisheries 

3/15/74 Chairman Sullivan, -jointly with 'Chairman Teaeue, Committee 
on Science and Technology, express interest in assessment 
of international shipping, including: 

1) Fundamental shipping technology 

2) Utilization of nuclear technology 

3) Airfoil techniques 

4) Fast sailing vessels 

**4/3/74 Chairman Sullivan requested assessments concerning: - 

**1) Exploration and exploitation of resources of Continental 

Shelf and deep seabed beyond 
**2) Fisheries 

3) Aquaculture and mariculture 

4) Oceans pollution - monitoring 

5) Oceans pollution - removal 

6) Submersible vessels 

7) Undersea habitats 



* Assessments currently being performed 
** Assessments planned for FY 1975 and 1976 



134 



9/18/74 Chairman Sullivan expresses interest in TAAC National 
Growth Policy Study and Committee's deep interest in 
long-term growth and its implications. 

3/7/75 Chairman Sullivan and Ranking Minority Member Ruppe request 
assessment of alternatives to unregulated ocean dumping and 
current status of related research and development. 

House Committee on Public" Works 

1/23/74 Chairman Blatnik suggests following areas of interest: 

1) National public investment policy and population 
distribution 

2) Transportation policy 

3) Water resources 

4) Pollution abatement 



House Coinmitt< 



n Science and Technology 



*•• 1/22/74 



3/15/74 



12/13/74 



Cli;: an Teague and Ranking Minority Member Mosher request 
ass iments concerning: 

1) Energy R 6, D 

2) Technology data bank 

3) Materials R&D 

4) Technology of unemployment 

Chairman Teague, jointly with Chairman Sullivan of the 
Committee on Merchant Marine and Fisheries, expresses 
strong interest in international shipping, including: 

1) Fundamental shipping technology 

2) Utilization of nuclear technology 

3) Airfoil techniques 

4) Fast sailing vessels 

Chairman Teague and Ranking Minority Member Mosher update 
of previous request of 1/22/74, with emphasis on assess- 
ments involving: 
* 1) Materials 

* a) information system 

* b) national stockpile 

* c) reuse 
#*2) Five-year r&d 



12/17/74 Chairman Teague and Ranking Minority Member Mosher re- 
quest, or emphasize urgency of, assessments involving: 

* 1) Planning and programming energy ' R&D - as contained 

in required ERDA reports. 

2) Identifying size of energy resources and unassociated 
uncertainties 

3) Materials technology and availability 
*4) Tracking energy studies and data 

* 5) Uncertainties which inhibit development of new energy 

technologies 



135 



House Committee on Ways and Means 

** 1/29/75 Chairman Ullman, jointly with Ranking Minority Member 
Schneebeli and Chairman Rostenkowski and Ranking 
Minority Member Pettis of the Subcoinmitte on Health, 
requests assessments concerning: 
** 1) Medical malpractice 
**2) Long-term medical care 
3) Adverse drug reactions 

Senate Committee on Aeronautical and Space Sciences 

* 1/21/74 Chairman Moss, acting for Senator Goldwater, requests 

assessment concerning solar energy. 

** 10/10/74' Chairman Moss requests assessment of means of determining 
research and development priorities. 

Senate Commit tee on Agriculture and Forrestry 

**6/25/74 Chairman Talmadge requests assessment of use of broadband 
two-way telecommunications in rural areas. 

Senate Committee on Appropriations 

* 2/15/74 Chairman McClellan, on behalf of Chairman Byrd and Ranking 

Minority Member Case of the Transportation Subcommittee, 
requests assessments concerning automation in urban rapid 
transit . 

* 9/27/74 Chairman McClellan, on behalf of Chairman Byrd and Ranking 

Minority Member Case of the Transportation Subcommittee, 
requests expansion of mass transit assessment. 

Senate Committee on Commerce 



1/14/74 Senator Hollings' National Oceans Policy Study requests 
assessments concerning: 

* 1) Impact of technology on coastal zone 
2) Ocean engineering technology 

** 3) Fisheries 

* 4) Deepwater ports 
** 5) Ocean mining 

6) Weather modification 

7) Acquaculture 

8) Oceans monitoring 



136 



* 1/24/74 Chairman Magnuson suggests assessments concerning: 

1) Energy savings in manufacturing 

2) Safety in disposal of nuclear waste 
* 3) Resource and energy recovery 

4) Upgrading railroad tracks 
••• 5) Oceans technology, including 
»* a) fisheries 
* b) deep" water ports 

6) Retrofitting offices and residences for energy savings 

7) Alternative energy sources for autos 

8) Mutagenic testing 

9) Detergents 

10) Predator poisons 

11) Population and conservation taxes 



2/11/74 



Senator Rollings' National Oceans Policy Study requests 
assessment on utilization of U.S. Continental Shelf. 



* 8/13/74 Chairman Magnuson, jointly with Chairman Rollings of Sub- 

committee on Oceans and Atmosphere, request assessment of: 
* 1) National growth policy 

*2) Projected effect of economic and demographic growth of 
coastal zone. 

10/10/74 Chairman Magnuson, acting for Chairman Tunney of the Sub- 
committee on Science, Technology and Commerce, requests 
assessment of computer managed technology 

' 10/17/74 Chairman Hart of the Subcommittee on the Environment re- 
quests assessment of cost and benefit of environmental pro- 
tection regulations and equipment on the automobile industry, 

* 1/15/75 Chairman Magnuson, acting for Senator Tunney, recommends 

assessment involving materials wastage. 

* 1/23/75 Chairman Magnuson, jointly with Chairman Jackson, Committee 

on Interior and Insular Affairs, requests assessment of 
feasibility of separating consideration of leasing for 
exploration from leasing for development and production 
on Outer Continental Shelf. 

?* 2/19/75 Chairman Magnuson, on behalf of Senator Tunney, requests 
comprehensive assessment of technology and world trade. 



3/20/75 Chairman Magnuson and Ranking minority Member Pearson, jointly 
with Senators Hartke and Weiker, Chairman and Ranking Minority 
Member of the Surface Transportation Subcommittee, request 
review of U.S. Railway Association plan for reorganization 
of rail service in the 17 state region covered by the Regional 
Rail Reorganization Act of 1973, and the issues it raises about 
the future of rail service in this region 



137 



| e Committ ee o n Foreign Rel ati ons 

* 9/19/74 Chairman Fulbright , acting for Chairman Muskie and Ranking 
Minority Member Case of the Subcommittee on Arms Control, 
International Law and Organization, requests an assessment of the 
the accuracy of DOD estimates of potential effects of limited 
nuclear warfare on U.S. society. 

Senate Committee- on F inane e 

2/27/75 Chairman Long, on behalf of Senator Talmadge, Chairman of the 
Subcommittee on Health, requests assessment of: 

1) Physician's fees 

2) Hospital Out-Patient Service 

Senate Commit tee 'on In terior and Insula r^Afj^-e 

*. 12/30/7/, ^airman Jackson indicates need for comprehensive assessment 
of energy technology and joins request of House Committee 



on Science and Technology for that purpos 



1/23/75 Chairman Jackson, jointly with Chairman Magnuson, Committee 
on Commerce, requests assessment of feasibility of separati, 
consideration of leasing for exploration from leasing for ' 
development and production on Outer Continental Shelf. 

Senate Co mmittee on Labor & Public Welfare 

**12/ll/74 Chairman Williams, jointly with Senator Javits and Congress- 
men Rogers, Nelsen, and Carter of the House Subcommittee on 
Public Health and Environment, Committee on Interstate and 
Foreign Commerce, request assessments in: 
**1) Spiraling costs of health care 
2) Uneven quality of health care 

** 2/6/75 Chairman Williams , on behalf of Chairman Kennedy and Ranking 
Minority Member Javits of the "Health Subcommittee , requests 
technology assessments on the following: 
**1) Cost and quality of clinical laboratories 

2) Medical record information requirements 
**3) Cost control studies, i.e., effect of regulation of 
price, effect of deductibles and coinsurance on ut- 
ilization of health care, efficacy of new technology 
and procedures, productivity measures, and cost of ad- 
ministering health insurance. 

Senate Committee on Public VJorks 

1/29/75 Chairman Randolph requests assessment of Federal assistance 
to energy and coal research facilities in Appalachia and 
West Virginia. 

Senate Select Committee on Nutrition and Human Needs 

** 2/7/75 Chairman McGovern asks OTA to determine whether protein is 
being wasted by being fed as grain to livestock, and if so, 
what government policy changes are necessary to remedy this 
waste. 



138 



Joint Economic Commit' tee 

9/10/74 Chairman Patman, acting for Chairman Bentsen of the 
Subcommittee on Economic Growth, requests assessment 
of industrial innovation in enhancing productivity and 
constraining prices with respect particularly to: 

1) Housing 

2) Power generation 

3) Transportation 

4) Basic metals and machine tools 

2/11/75 Chairman Humphrey requests assessment of feasibility of im- 
proving and enlarging defense research and production facilities 

'■'■' 2/28/75 Chairman Humphrey and Representative Reuss, Chairman of the 
Subcommittee on International Economics, request a compre- 
hensive study of technology and world trade with assessments 
of policy options to strengthen international trade position. 



Technology Assessment Board 

*l/22/74 Senator Humphrey suggests assessments concerning: 
*1) Agricultural information 

a) use of ERT satellites in agriculture 

2) Fertilizer 

3) Food processing and storage 

4) Farm-to-market roads 

* 9/19/74 Congressman Udall suggests assessment of a proposed 

National Growth Policy. 

* 11/6/74 Senator Stevens suggests topic of accessibility of 

mineral resources for assessment. 

* 11/7/74 Senator Humphrey suggests assessment of priorities in 

federally supported research and development. 

2/26/74 Senator Schweiker requests review of U.S. Railway Association's 
conrail plan. - ; 



Other Members of the Congress 

1/7/74 Congressman Wyman recommends for consideration Federally- 
assisted private corporations to develop a commercially 
feasible process for. making gas and oil from coal. 

1/28/74 Senator Hartke proposes consideration of impact of tech- 
nology on future growth and development of the Nation. 

1/28/74 Congressman Vanik requests data concerning reduced energy 
consumption attributable to winter Daylight Savings Time. 

1/29/74 Congressman Perkins suggests the problem of outdated statis- 
tical information as area of concern. 



139 



2/14/74 Congressman Whit ten, making no specific request, callu 
attention to work in the following areas: 

1) Food and Drug assessment of need for Delaney clause 

2) Study of future lab needs for Food and Drug 

3) Appropriation to National Academy of Sciences to 
evaluate programs of Environmental Protection Agency. 



11/21/74 Congressman Aspin suggests as area for 
gers involved in use of aerosols. 



issessment the dan- 



12/3/74 



Congressman Pickle requests study of dangers imposed by 
use of freon in aerosol cans. 



2/10/75 Congressman Wolfe requests analysis of study made on atmosphci 
effects of a fleet of supersonic transports. 



140 




li s I § ztz. i:- «*| 123*1.* Sfri-f 



j ss -°- ^ 5" 
glfll «* 

S-S. 8.--C ;5j 



I! 1 Ilfa,- 




. Hi ifffill If! HI 

mm 
III! 

Hlillli 




141 



JlPllPi! 




11 



mj 



mil !i! v 







1 3| fiHilSitl|||i!?jrj! 

m\ mil 

S-S£-SSXJ * 




Htt Si l 3 

lill &i! 

ml iJlKi 

iicti-iiiii 

Jilirliril' 






§ 3a4 9-1 j J* £ 

llfli II r 

o 3>2.2-§ c *■;£ -6. 

Hlllll f 

! m #? til 

! aMBw 



!i 




i gj t|ililS|l||li| 

1"8 :^3-g.-g|'Sl co '2,l3 




6 3 1 4#fpfii Pi 







ii 

ifr] |: 



kJ 



HP Bill 



£5J 5 



, pNl? 




lillililllfilllllSj 



Id jots 

s st sgfiii 
ill] ?3 r& 



142 



£ l-§§ <3&* 

"" S- t" JS*.2JS 

J 11 Jilt 

I i« III! 



1 3PII 1 




2 .§ 



1*5*111 



I'E* §|| 

Iliiws (hi 

%«5HJhu 



Inif ihiitlllilH 



Sal** 



-9 II 



illiiiiir 



5 i*j 

ml 







m in j^i 

111 *« 5 Hi 
s 111 1M s g^ 

I tfsa Si* ^JSl 
* -<:i6 ■ -s «• * "s- E 

3 3?l ]>{ l»j 

Hi HI ft? 

ill ti f|| 

t i|i -iii i*i 
i **\ *Vi W 

a fil *SS 5*1 

1 



143 
October 13, 1972 - 7 - Pub. Law 92-484 



t6 STAT. *03 



Sac. 12. (e) To enable the Oflce to carry out ita power* end duties, 
there ie hereby authorised to be appropriated to the OAce, out of any 
v in the Treasury not otherwise expropriated, not to exceed 
),000 in the aggregate for the two fiscal yean ending Jane 30, 
, end Jnne 80, 1974, and thereafter wen sum* a* may be nec e ssary, 
(b ) Appropriation* made pursuant to the authority provided in 
•ubeectkm (a) ahall remain evailable for obligation, for expendi- 
ture, or for obligation and expenditure for each period or period* ee 
may be epednedin the Act making eneh appropriation* 
Approved October 13, 1972. 



LtGISUTIVI HIS TORY I 

HOUSE REPORTS i *. 92 -4W (com. ee Science and Astrenautlea) and 

Re. 92-1436 (Cam. of Conference). 
SENATE REPORT No. 92-1123 (Cobb, on Role* and Ada IMrt rati ea). 
CORORESSIOMAL RECORD, fel. 11* (1872). u»-cra*ienj. 

Peb. B, oencldered and paased House. 

Sept. 14, eonaldered and paaaed Senate, aaaaded. 

Sept. 22, Senate agreed to conference report. 

Oct. 4, Houcc agreed to conference report. 



144 



October 13, 1972 - 7 - Pub. Law 92-484 



86 STAT. 803 



APPROPRIATIONS 

Sec. 12. (a) To enable the Office to carry out its powers and duties, 
there is hereby authorized to be appropriated to the Office, out of any 
money in the Treasury not otherwise appropriated, not to exceed 
$5,000,000 in the aggregate for the two fiscal years ending June 30, 
1973, and June 30, 1974, and thereafter such sums as may be necessary. 

(b) Appropriations made pursuant to the authority provided in 
subsection (a) shall remain available for obligation, for expendi- 
ture, or for obligation and expenditure for such period or periods as 
may be specified in the Act making such appropriations. 

Approved October 13, 1972. 



LEGISLATIVE HISTORY ; 

HOUSE REPORTS* No. 92-469 (Conn, on Scienoe and Astronautics) and 

No. 92-1436 (Conn, of Conferenoe). 
SENATE REPORT No. 92-1123 (Conn, on Rules and Administration). 
CONGRESSIONAL RECORD, Vol. 118 (1972) i 

Feb. 8, considered and passed House. 

Sept. 14, oonsidered and passed Senate, amended. 

Sept. 22, Senate agreed to oonferenoe report. 

Oot, 4, House agreed to oonferenoe report. 



145 



DISCUSSION 

Several issues were raised after Dr. Thomas' presentation. It was 
noted that both its legislative mandate and legislative exigencies require 
the OTA's work to be especially timely and responsive to the Congress. In 

addition, the OTA maintains very close liaison with committees both to 

receive suggestions for assessments, and to brief committee staff through- 
out the process of the assessment, making available working documents and 
monthly reports submitted by contractors. 

Mr. Thomas also noted that the OTA does two types of assessments 

both long and short range. Short range or "mini-assessments" are done on 
an urgent basis by the exploratory assessment section of the Office. For 
example the OTA recently completed a two week assessment of the impact of 
the ERDA budget. The exploratory assessment section serves another function 
in addition to that of providing rapid responses on assessment questions. 
It performs exploratory assessments to determine whether an in-depth. long- 
range assessment is warranted. 

A question was raised about the possible similarities between technology 
assessment research and science policy research. Dr. Thomas remarked that 
the two clearly are distinct. Technology assessments generate information 
to be used in policy research or policy deliberations. As an example he 
noted that the assessment of materials information system is designed to 
generate information to indicate where such information might be used most 
appropriately in policy research. 



146 



["OVERSIGHT RESPONSIBILITIES AND ACTIVITIES OF THE SENATE COMMITTEE ON 

GOVERNMENT OPERATIONS AND SEVERAL OF ITS SUBCOMMITTEES,"] STAFF OF THE 

SENATE COMMITTEE ON GOVERNMENT OPERATIONS: STATEMENTS BY MARILYN 

HARRIS, HOWARD FELDMAN, JIM DAVIDSON, BILL GOODWIN, VIC REINEMER 

AND RON CHIODO 



Ms. Marilyn Harris 

I believe Mr. Beckman says it well. Our Committee has a tremendous 
challenge which is the oversight of the agencies of the government, espe- 
cially with respect to their economy and efficiency. As you know Senator 
Ervin was Chairman of the Government Operations Committee until his re- 
tirement. This year, Senator Ribicoff became Chairman. The committee 
reorganized its subcommittees and changed some jurisdictions with the 
hope, naturally, of making our Committee as efficient as we would like 
the Federal Government to be also. As part of our organization, we estab- 
lished the Oversight Procedures Subcommittee, about which Mr. Goodwin will 
be speaking. One Subcommittee was abolished, Reorganization, Research and 
International Organizations and that jurisdiction was taken to the full Com- 
mittee. Our Committee is organized into unique Subcommittees. Each Sub- 
committee has its own budget and is funded under a separate section of a 
General Funding Resolution. Because of this type of organization, there 
is quite a bit of autonomy within our subcommittees, whereas in some other 
committees, there may not be as much autonomy because of the type of com- 
mittee structure established. 

I wanted to introduce the panel to you, and I thought the best way for 
you to achieve some understanding of our Committee and its Subcommittees is 
for each person who is representing that subcommittee to tell you specifically 
what it is doing. So I will introduce the panel to you with that objective 
in mind. 



147 



On my left is Jim Davidson, Counsel to the Intergovernmental Relations 
Subcommittee. That Subcommittee investigates the intergovernmental rela- 
tions between the U.S. and the states and municipalities, including the 
fiscal relationship between the state, local and Federal governments. Jim 
will tell you more when we begin. On my immediate right is Mr. Howard 
Feldman, who is the General Counsel to the Permanent Subcommittee on In- 
vestigations. This Subcommittee is non-legislative and is concerned strictly 
with investigation and oversight. And to his right is Mr. Bill Goodwin the 
staff director of our newly created Subcommittee, the Oversight Procedures 
Subcommittee. One of its main goals will be to develop methods to improve 
how the Government Operations Committee and other committees can develop 
procedures for improving oversight. Also, Vic Reinemer, staff director of 
the Reports, Accounting and Management Subcommittee* Senator Metcalf that 
Subcommittee's Chairman, was one of the Senators who, I believe, generated 
the initiation of this Conference. Last year, the Reports Subcommittee 
was titled the Budgeting and Management Subcommittee. Some of that Sub- 
committee's jurisdiction was changed. It is concerned with the accounting, 
financial reporting and auditing arrangements and the development of manage- 
ment capability and efficiency of the Federal Government. It is also con- 
cerned with Federal advisory committees, Federal reports, questionnaires of the 
Federal Government and interrogatories. On the very end is Mr. Ron Chiodo, 
a counsel from the Subcommittee on Federal Spending Practices, Efficiency 
and Open Government. This Subcommittee is chaired by Senator Chiles. Last 
year it was called the Subcommittee or the Ad Hoc Subcommittee on Procure- 
ment, and has now been changed to a permanent subcommittee. 

I think we will begin. Mr. Feldman. 



148 



Mr. Howard Feldman 

Thank you. I think our Subcommittee is probably unique. I must take 
issue with the comment that we don't have a legislative purpose. Actually 
we had legislation that came out last week as a result of one of our in- 
vestigations. This involved a study of the purchase of jet fuel by the 
Department of Defense. We compared their costs to the airlines and found 
huge overpayments. Previously we had done an investigation on the cut-off 
of oil to our military, especially the 6th fleet, after the 1973 Arab- 
Israeli war. Senator Jackson, Chairman of the Subcommittee , introduced legis- 
lation providing for criminal penalties in the event any individual or corpora- 
tion willfully cuts-off oil to the military. But Marilyn is correct when 
she says that our responsibility is not that of developing legislation, 
holding subcommittee hearings on legislation and moving it to the full com- 
mittee on a regular basis. We hope that our investigations result in appro- 
priate remedial legislation. Two weeks ago, for example, Senator Jackson 
introduced an amendment to Senator Williams' securities bill. That amend- 
ment gave the SEC the authority to require the validation of securities and 
the authentication of securities. This was based on a three year investiga- 
tion on stolen securities, organized crime, movement through brokerage 
houses, banks etc. The bill was passed by the Senate and is now in con- 
ference. We consider it a great accomplishment after several years of in- 
vestigation. 

The Permanent Subcommittee on Investigation has a long and glorious 
history. It previously was known as the McClellan Committee and during 
that time we went through labor racketeering, the Valachi hearings, TFX 
and others. Prior to that it was the McCarthy Committee, and I have great 
reservations about some of the techniques and procedures used at that time. 



149 



But as a result of the McCarthy days we have very rigid procedures and 
guidelines governing our hearings — all of which, by the way, are held 
under oath — about rights to counsel, about subpoenaing documents and 
other safeguards. We run what I term investigative rather than oversight 
hearings. Seldom do we have anyone before our Subcommittee who has not been 
interviewed, whose records we have not supoenaed or from whom we have not 
received some corroborating information about the investigation. It takes 
many months to put on some of our most extensive hearings, although we have 
put on less extensive sessions also. 

We deal quite regularly with people's lives and reputations, and I 
find it a very heavy responsibility. For example, when we put on our se- 
curities hearings, some witnesses would mention a hundred names who were 
involved in illegal transactions. We had to go out and try to run down 
those people in the event their names were going to be mentioned. It is 
tedious work. We have a large staff, and some talented veteran investiga- 
tors. That's another question. How do you pick a good investigator? It's 
very, very difficult. They can be ex-FBI people, they can be newsmen, they 
can be attorneys or others. The prerequisite is to have a knack to ferret 
out the facts. 

We've done such things as the Russian grain deal, cut-off of oil to 
the military, the CHAMPUS investigations (that's the military spending for 
psychiatric care of benefits for children), various other oil investigations, 
the Russian grain deal II (as we term it - the October sale), stolen se- 
curities cases, material shortages hearings that were more of an oversight 
nature with Senator Huddleston, when he was on our subcommittee handled 
those. At present, we are involved in an investigation of DEA (the Drug 



150 



Enforcement Administration), focusing on their efficiency and operations. 
Our jurisdiction is very broad - and I should point here that our Subcom- 
mittee is in turn a combination of two subcommittees which were combined 
in March of 1973: the Permanent Subcommittee on Investigations and the 
old National Security Subcommittee, which was chaired by Senator Jackson. 
That Subcommittee has looked into many national security questions with a 
very talented group of people who are leading experts on foreign policy. 
Thus, we have a dual jurisdiction - domestic and foreign. The resolution is 
extremely broad and it permits us to function in just about any area. We 
have cultivated relationships with state and local police and with other 
governmental agencies. Our files have become a repository for many scholars 
who want to write books on racketeering and organized crime and things of 
that nature. We've tried to maintain a more balanced investigative ap- 
proach, and I think it 's exhibited in the kinds of things we are doing now. 
I could go on and on regarding our investigations, but I don't want to take 
up your time. We coordinate with all the other subcommittees. We've been 
looking at utility questions, for example, and Vic is the expert here. We 
hope to build on things that he's done. So with that I will depart. 

Does anyone have any questions for Mr. Feldman concerning his Subcommit- 
tee? Next is Mr. Jim Davidson, Coune«l to Intergovernmental Relations Sub- 
committee, which is chaired by Senator Muskie. 

Mr. Jim Davidson 

As Marilyn explained the Intergovernmental Relations Subcommittee 
has principal responsibility for the relationships between the Federal, 
State and local governments, funding assistance programs, to the extent 
that it does not traditionally fall within Judiciary Committee concerns 



151 



for Federalism and constitutional problems of proper Federal and state 
roles for administering or financing programs. Finally we explore general 
legislative problems where Federal legislation has an import on the opera- 
tion of state and local governments apart from financial problems. 

In addition, our Subcommittee , through mutual agreement between the 
Chairman of the Judiciary and the Government Operations Committees, has 
inherited a certain role for examining government policies with respect to 
information in the Freedom of Information Act, the Privacy Act, and the rights 
of members of Congress and congressional committees to information in the 
possession of the Executive Branch which in the last Congress produced the 
so-called right to information or executive privilege bill which was passed 
by the Senate but was not adopted by the House. I think this will give you 
an idea of the types of oversight roles that we have fulfilled during the 
last Congress. We did extensive hearings and investigations of the General 
Revenue Sharing program in June of last year when we held four-days of 
hearings oversight on the implementation of the program. We are attempting 
to examine the limited prescriptions on the use of revenue sharing funds, 
how the funds were being used, whether or not they were being used in 
violation of any provision of the Revenue Sharing Act with emphasis on 
the civil rights provisions of that Act. Last years hearings involved 
the city of Chicago and its use of revenue sharing funds in support of the 
police department work with focus of attention there. Those hearings and 
ones yet to come will figure strongly in this year's congressional con- 
sideration of the renewal of revenue sharing. 

Also last year we focused on the impact of the energy crisis on state 
and local governments. This was a financial as well as an administrative 



152 



problem, looking at the cost of state and local governments of dealing with 
shortages of energy as well as the impact on constituents of those jurisdic- 
tions and the problems they were facing with State and local administration 
of Federal energy programs. That investigation required an extensive sur- 
vey of all states and their energy offices and how they were implementing 
Federal and state energy programs. That resulted in a staff study that was 
published in January of this year. The study also drew from hearings that 
were held both last June and last October on the Federal energy crisis and 
its impact on state and local governments. 

As I also mentioned we have tried to examine the relationship and the 
impact of the concept of new Federalism when this was a central theme of the 
Nixon Administration — that is how State and local governments were attempting 
to deal with the decentralization of the administration of Federal programs 
as greater emphasis was being placed on the regional offices. That included 
the questions of how this reorientation was actually working - was the shift 
of responsibility going out to the regional off ices, or are they simply still 
figureheads; and was in fact the decision for state and local programs still 
being made in Washington? We had eight days of hearings on the new Federalism 
program bringing in the secretaries of HEW, HUD and the Director of the Of- 
fice of Management and Budget. 

The Subcommittee, also ir 
level of public confidence and concern about Federal Government, State govern- 
ments, and local governments and the leadership and administration of programs 
at each of those levels of government. The Subcommittee retained the Louis 
Harris organization to conduct a poll of American citizens and also more than 
206 local and state officials were interviewed during this process - both 
in person and through written questionnaires attempting to evaluate what the 



153 



public" 8 perception was of the administration of local, State and Federal 
programs and in turn what officials at those levels of government thought 
the public's perception was of the administration of government programs. 
That produced an interesting split in the evaluation. There was a much 
greater feeling that a better job was being done by those holding office 
than by those who elected them. That study was published in December of 
1973 in 3-volumes and still used as a basis of evaluating public perceptions 
of local, State and Federal programs. 

In the area of constitutionally related questions of the Congressional 
right to information as well as freedom of information and privacy, our 
Subcommittee played a principal role in developing certain amendments to 
the Freedom of Information Act in conjunction with the Judiciary Committee. 
Also we conducted with the Judiciary Committee a series of hearings in 1973 
and followed-up with hearings last year into questions of government se- 
crecy, the withholding of information both from the public and from the 
Congress and the related constitutional questions involved as I mentioned 
before, that produced the so-called Congressional Right to Information 
Act which was enacted by the Senate and which will be reintroduced again 
by our Subcommittee this year. In the area of government secrecy - 
we held in 1973 a total of 7 days of hearings. Last year we conducted 
5 days of hearings into the Executive Branch administration of Executive 
Order 11652 which was the most recent in a series of Presidential Execu- 
tive Orders prescribing the procedures for classifying government secrets 
confidential, secret and top secret. We hope that another round of 
hearings this year will help produce Congressional guidelines for the 
classification of government secrets which will replace the Executive 
Order guidelines for classification. 



154 



The Subcommittee through the full committee played a major role in 
the development of the Privacy Act of 1974 and this year is working toward 
the implementation of that Act which goes into effect September 27th. We 
are working almost on a daily basis with the Office of Management and 
Budget in the development of guidelines for the implementation of the Act, 
and plan hearings in the very near future on oversight of the level of 
preparedness of various agencies toward implementing the Privacy Act. 

One incidental role of the Subcommittee which will probably be con- 
tinued in this Congress involved hearings on the Congressional oversight 
of U.S. intelligence agencies. In December of last year we had two days 
of hearings and had planned follow-up hearings in January and February 
until the N.Y. Times broke the story of intelligence agency surveillance 
of U.S. citizens. It was our Subcommittee that developed the resolution 
which the Senate adopted establishing a Senate Select Committee on In- 
telligence Oversight and will probably receive again the referral of any 
recommendation by that Select Committee for the establishment of a per- 
manent Oversight Committee on Intelligence. As you can see, our Sub- 
committee has rather diverse responsibilities, but part of it is by 
tradition and part of it is just by agreement between the Chairmen of 
the Government Operations Committee and Judiciary. I will defer to 
others if you have any questions. 

Thank you. Bill Goodwin is staff director of the Oversight Sub- 
committee, the new Subcommittee which I mentioned previously. As you 
may know, the proceedings of this conference or seminar are going to be 
published in a document which his Subcommittee will be preparing. 



155 



Mr. Bill Goodwin 

Well very briefly, our Subcommittee was created at the first of this 
year. 

We have four primary duties. First, the development of better methods 
and procedures of congressional oversight. Second, the study of the utili- 
zation and disposal of Federal property. Third, the consideration of budget 
measures other than those within the jurisdiction of the Budget Committee. 
Fourth, general oversight of the work of the Federal Paperwork Commission, 
which was created at the end of last year. Of these our work in oversight 
is our major responsibility. 

One of the chief reasons that our Subcommittee was created was the 
general feeling that Congress can do a better job of oversight. As 
Walter said at the beginning of this seminar, the climate is right for im- 
proving our oversight functions. You might say that the question is 
whether we are going to have oversight by scandal — do we have to create 
a select committee every time we have a Watergate or a revelation in the 
N.Y. Times about C.I. A. activities. 

The House, I think, got the jump on the Senate with its committee 
reforms and the House Government Operations Committee now has new 
strength and responsibilities in the oversight area. 

Our Sttbcommittee is planning to survey the Senate committees, 
which will be similar to the recent report of the House Government Opera- 
tions Committee — although we do not have the authority by Senate rule 
that the House committee has by House rule at the present time. 

We also intend to put together in committee print form a fairly compre- 
hensive reference work on oversight sometime within the next two months. 
This will be a document, which we intend to make available to all members 



156 



and all committees giving outlining the different types and different 
techniques of oversight. In effect, we will take this seminar and ex- 
pand it into more technical avenues. 

We also intend to initiate studies of our own in oversight, and 
to use the resources of the General Accounting Office and of CRS. We hope 
to be able to serve as a coordinator, and to more or less advertise the 
work, that is going on in CRS and in GAO. 

Our primary function is to work on better committee functions of the 
oversight process and to come up with ideas of how the Senate committees 
can better oversee the operations of the executive branch on a day-to-day 
and month-to-month basis as opppsed to waiting for some scandal to break 
in the Washington Post . It's not going to be easy, but budget reform 
wasn't easy, either. As I recall, Vic, it took 2 1/2 to 3 years to get 
that bill through the Congress - so we don't visualize results overnight. 

In the meantime, our Subcommittee will be doing some oversight of its 
own, especially of GSA as it pertains to the use and disposal of Federal 
property — not the procurement of property. The Subcommittee recently 
staffed oversight hearings in the full Government Operations Committee on 
energy conservation to try to get a grasp of what all the agencies are 
doing in energy conservation and what sort of coordination is taking 
place in the Executive Branch. We also will be watching the Federal 
Paperwork Commission and the National Commission on Productivity and 
Work Quality which should be reformed within the next 2 or 3 months into 
a National Center for Productivity. 

Those are some of the Subcommittee's own oversight plans. 

The staff hopes to be of help to everybody, frankly. We are not 
going dabbling into other people's jurisdiction. We want to put together 



157 



information and ready references that will be helpful to all members of 
the Congress and their staffs and all the committees and their staffs 
and to proceed with the development of better methods by which Congress 
can do its oversight job. 
Ms. Marilyn Harris 

Next, Mr. Vic Reinemer, the staff director for the Reports, Accounting, 
and Management Subcommittee. 
Mr. Vic Reinemer 

I see the House Government Operations Committee staff came in so 
please impose their five minute fule on me if I go beyond, so that they 
have plenty of time. 

I would like to just mention just two of our oversight activities 
because they involve some different techniques which may be of interest 
to you. They also both afford examples of something which we always find 
in evaluation which makes our job more difficult, which I think we ought 
to talk about briefly. And that's the congressional compulsion to over- 
legislate. It's part of the process. The members see a situation that 
needs correcting. They say there ought to be a law, let's put out a press 
release, and put it on through. And then we get really this extraordinary 
confusion of existing laws, sometimes conflicting laws. So when we do try 
to evaluate or make a particular program work, instead of going back and 
using a much better law — and here the 73rd and 74th Congresses offer 
some beautiful precedents — instead we try to legislate some more. 

This is what is going on right now in the energy field, especially re: 
foreign ownership of energy corporations and other corporations. Instead 
of working with the various regulatory agencies, which have all sorts of 
authority to collect and tabulate and make meaningful reports on ownership 
of companies foreign or otherwise, we are passing more laws, and not paying 
as much attention as we all should in my opinion to the development of the 



158 



regulations and the enforcement of the various reporting require- 
ments. 

I was so glad to hear Ellen Collier earlier in the day talk about 
the importance of the agency reports. But what I am talking about here 
is the corporate reports to the regulatory commissions, and the information 
within them, which is so important to enforcement of laws whether it be 
anti-trust, conflict of interest or a rule-making proceeding or whatever. 
So I just wanted to make this point — that maybe we will have more peo- 
ple like Senator Barkley, who made quite a point of not introducing legis- 
lation — except occassionally something having to do just with Kentucky. 
He prided himself on being a developer of particular refinements of legis- 
lation rather than being the first to put his name on the bill. 

Well, anyway, when we dealt with this matter of corporate disclosure 
our technique was first the traditional study, in which we obtained the 
cooperation of CRS and GAO, then came the hearings at which it became a- 
bundantly clear that the various regulatory commissions had all the 
powers they needed to cc'lect information as to who owned and controlled 
various companies that report to the regulatory commission. We also 
saw that the regulations had just grown up topsy turvy. 

So as a result of these hearings where we established that we didn't 
need laws we — and here the GAO was instrumental — established a steering 
committee on uniform corporate reporting requirements. And now when they 
go out for rule-making comes the real follow-through. For example the ICC 
has formally proposed adoption of these model corporate reporting re- 
quirements that deal with the stock-holdings, debt-holdings, corporate 
structure and affiliations of officers and directors, of major regulated 
companies. If the ICC and the other commissions adopt these requirements 
we are going to have, all of us, for the first time, a great deal of 



159 



current information which has otherwise been obtainable partially only 
by special and time-consuming proceedings. But the difficulty here is 
that what adoption of the model disclosure regulations requires is for 
various members and interest groups to actually become parties in ad- 
ministrative proceedings. So this is one thing that Senator Metcalf and 
a few others, especially on the House side too, are increasingly doing — 
being formal parties to administrative proceedings of the regulatory 
commissions. The other activity that I want to mention briefly has to 
do with the advisory committees. This Federal Advisory Committee Act is 
two years old now, two years and 4 months. And actually the number of 
Federal advisory committees has decreased from 1439 to — OMB and GSA 
have a different count, but Jerry Sturgis here has the best count, 

what is your figure Jerry 1242. So we have gotten rid of about 200. 

And that is a net reduction. Last year Congress established 53 new 
Federal advisory committees, by statute, despite the very explicit 
language in our two-year-old statute that no new Congressional statutory 
advisory committees shall be established unless there is proven that 
there isn't another one that can do the job or perhaps the agency it- 
self could do the job. 

We really are looking for some help on how to handle this one. What 
we did about a year and a half ago was to send to all the standing com- 
mittees a list of all the advisory committees which are subject to their 
jurisdiction. Now we are going to send out a little stronger document 
this time. It will include a list of all the new advisory committees 
that they established, probably not in accordance with the law which 
had been resoundingly approved. Here is the sort of an area where we 
are going to continue in and to ask the CRS for some assistance. It 
has been so helpful on indexing. We will soon have available the 1975 



160 



report on all the advisory committees. This will include the membership, 
meeting information and all that. That's already available on microfilm. 
And by Labor Day we will have an alphabetized index, as we did after the 
first report came out two years ago. This will be alphabetized both as to 
institution and individual. It will be a very valuable resource. 

Ms. Marilyn Harris 

Thank you, Vic. I would like Ron Chiodo to share with you the re- 
sponsibility of the Federal spending practices, Efficiency and Open Govern- 
ment Subcommittee, chaired by Senator Chiles. 

Mr. Ron Chiodo 

Ours also is a relatively new Subcommittee. In 1969 by law the 
Commission on Government Procurement was formed to review the procurement 
activities of the Federal Government and recommend any necessary reforms. 
In 1972 their study was completed and included 149 recommendations. As 
a result the Ad Hoc Subcommittee on Procurement was established in the 
Government Operations Committee last year to carry out the Commission's 
recommendations. This year the jurisdiction has been broadened, and now 
includes Federal spending practices, efficiency and open government. 

Very briefly, the Federal spending practices area includes Federal 
procurement and the laws, regulations and procedures governing Federal 
contracts, grants, transfer payments, and other spending arrangements 
includes the Office of Federal Procurement Policy, which was established 
just this year. It also includes all Executive Branch organizations 
which are responsible for Federal spending practices. To the extent that 
"Federal spending practices" doesn't cover the ball park, we have 
"efficiency" which includes the efficiency and economy of Federal spending 
practices generally. Finally, of course, we have "open government" and 
that particularly has to do with legislation regarding the public con- 



161 



duct of meetings of all branches of the Government. 

Just to give you a few examples, in the Federal Spending Practices 
area we are in the process of preparing for and will be conducting over- 
sight hearings with regard to how the Federal Government goes about 
procuring major systems, be they weapons systems or civil systems. The 
Department of Defense has procurement programs today which will total 
about $140 billion when acquisition is complete. The dollar figures 
involved are very high and the cost overruns that we've all noted in the 
newspapers contributes significantly to this. There is some concern that 
there's a lack of competition in the acquisition of major programs. We 
also see the problems of interservice rivalry that have to be contended 
with. In the civil area, GAO has done a recent report on civil acqui- 
sitions and has found some projects whose cost overruns have been as high as 
200%. So not all the problems are in the area of the Department of Defense. 
Continuing in this one area of oversight , we also have concern over the 
way the Federal Government acquires all kinds of property, be it major 
systems or perhaps just desks and typewriters. 

Today the acquisition process for the government involves something 
over 4,000 laws that are sometimes conflicting - certainly not all in 
one place. There is a need to revise and streamline the acquisition 
process generally. We are now working on legislation which would do 
that - bring the procurement laws together in a consolidated acquisition 
statute and amend the existing laws as is necessary to develop a con- 
sistent approach to the acquisition of property. 

By calling ourselves Subcommittee on Federal Spending Practices 
rather than just Procurement as was the ease last year, we also have clear 
jurisdiction over the grants area and we are conducting investigations 
with regard to grant programs, HEW comes to mind here. The dollar figures 



162 



involved again are very high, upward of $50 billion. We see the same 
thing I think you all have noted, that in some cases the process by 
which grants are let can be considered very confused. There have been 
many examples in which very little discipline has been applied to the 
system. There have been some isolated examples where awards have cer- 
tainly wired to favored recipients, perhaps suggesting fraud. We have 
introduced some legislation with regard to the grants area to help 
clarify the circumstances under which a grant should apply and the cir- 
cumstances under which instead a contract should apply. 

With regard to the second area of jurisdiction, that of the effi- 
ciency of Federal programs, we have been looking at the social security 
program as one example, and the food stamp program as another example. 
Questions that are being raised are primarily with regard to the mechanism 
that's being employed. Are applicants getting a fair shake out of the 
system? Is the system too overloaded with bureaucracy? Is there a 
better mechanism for providing the same benefits? In no case does 
our jurisdiction, though, extend to the objectives whether we should 
have a food stamp program or social security program. We would only 
raise questions as to whether the program was being conducted as effi- 
ciently as possible. 

The last area of jurisdiction of open government is rather obvious. 
There is legislation which has been introduced this year called Sun- 
shine legislation. (Audience: What state originated the basic legis- 
lation?) Ron: State of Florida. Florida was one of the first states 
that had pursued legislation to open its government meetings to the public 
and the term "sunshine" has been applied to legislation of this type. 
S. 5, a sunshine bill, is tentatively scheduled for mark-up this month. 



163 



This briefly covers our area of legislative oversight. We also have 
the typical activities, case work load and investigations that come up 
when particular violations of law are found just like many of the other 
subcommittees do. I will turn it back to Marilyn if there are any ques- 
tions. Marilyn: Are there any overall questions on the Senate side? 

Audience 

Just one, Mr. Goodwin didn't say who the members of his committee are 
and I would be interested 

Bill 

The Chairman is Senator Nunn. Other majority members are Senators 
Jackson, Muskie, Allen, and Chiles. The ranking minority member is 
Senator Percy and Senator Roth. 



164 



DAY II: CONGRESSIONAL COMMITTEE CASE STUDIES OF EVALUATION 
AND OVERSIGHT: SUMMARY OF PROCEEDINGS 



Committee staff experiences in doing or using various types of 

evaluation, quantitative evaluation research, and oversight formed the 

topic for the second day of the seminar, May 12. 

A. Oversight and Evaluation in the House Committee on International 
Relations: On-site Staff Visits 

The proceedings commenced with a presentation hy Jack Sullivan, staff 
Consultant to the House Committee on International Relations (formerly 
the House Committee on Foreign Affairs). Dr. Sullivan began his presentation 
by outlining the history of oversight activities in the International 
Relations Committee. Then he described the Committee's current sub- 
committee structure. In response to H. Res. 988, the Committee Reform 
Amendments of 1974, the Committee was given new jurisdictional responsibili- 
ties. It created three new oversight Subcommittees to deal with existing and 
new responsibilities. These are: (1) the Subcommittee on Oversight, which 
deals with oversight of foreign intelligence, administration of foreign affairs 
external research, and all areas not covered by other subcommittees; (2) the 
Subcommittee on Investigations, which deals with special foreign- policy- re- 
lated investigations referred by the chairman with the consent of the com- 
mittee; and (3) the Subcommittee on Future Foreign Policy Research and 
Development. 

Most of Dr. Sullivan's discussion focused on how the Committee and 
Subcommittees have gathered information for oversight of overseas operations 
of the U.S. Government. The Committee uses several techniques, including 



165 



onaite visits and related analysis by neutral bodies, such as the CRS 
or the CAO, and visits by members of Congress and committee staff investiga- 
tors. Dr. Sullivan emphasized the importance to both the International' 
Relations and the Foreign Relations Committees of onsite overseas investi- 
gative staff visits. 

He gave several case study examples to illustrate the utility of such 
visits as" an evaluation and oversight information gathering technique. For 
example, while making an onsite visit in Asia, Committee staff investiga- 
tors who interviewed local AID recipients learned that many women in Indonesia 
and the Phillipines using AID-supplied' birth control pills were suffering a 
variety of side-effects. AID headquarters and the chief AID physician 
discounted the importance and severity of these effects. They were in- 
terested in evaluating the success of the AID birth control distribution 
program in terms of numbers of persons participating (a process evaluation 
question) , rather than also in terms of the impacts of the program on the 
participants. After Committee staff reported back, to the full Committee, 
congressional pressure forced local AID administrators to investigate and 
confirm that women were indeed suffering side-effects. Additional onsite 
investigation indicated that due to GSA procurement regulations, each year, 
on the basis of lowest bid, a different pharmaceutical firm supplied AID 
with birth control pills. The chemical composition of the pills varied 
significantly enough to casue side-effects in women who were given new 
types of pills every fiscal year. Finally, congressional pressure and 
publicity caused the AID physician to change his position and the GSA 
to change procurement regulations so that women could receive the same 
drug continuously. 



166 



Dr. Sullivan indicated that onsite visits are only one important 
form of information-gathering for evaluation and oversight. However, 
he stressed that they are especially useful because they generate 
information that cannot be obtained by any other method including the 
use of neutral research bodies. Onsite committee staff investigators, 
unlike those of neutral research bodies, have immediate access to de- 
cisionmakers: that is, members of Congress. It is important, Dr. Sullivan 
stated, for investigators to have daily close contact, trust, and rapport 
with members. This relationship aids immeasurably in providing access for 
using evaluation and oversight information in policymaking. 

Dr. Sullivan emphasized, however, that two important criteria must 
be met to insure the success of onsite overseas staff investigations: 
(1) the investigators must be highly trained professionals, and (2) visits 
should be made in pairs, to aid the committee in corroborating evidence 
and statements made in interviews which Federal agency administrators 
might deny when testifying before the Congress. 

B. Oversight and Evaluation in the House Committee on Government 
Operations: The Freedom of Information Act 

William G. Phillips, who from 1971 to 1975 was staff director 

of the Foreign Operations and Government Information Subcommittee of 

the House Government Operations Committee, gave the second presentation 

of the afternoon. 



167 



Mr. Phillips began his statement by providing a detailed descrip- 
tion of the sequence of oversight and investigative activities and 
then legislative action which led to congressional passage and 
overriding of a presidential veto of amendments to the Freedom of In- 
formation Act of 1966, P.L. 89-487. The act provides, in general, 
that '"any person' might have access (with certain exceptions) to 
rules, opinions, orders, and other types of public records of various 
agencies of the Executive branch of the Federal government." 

Mr. Phillips noted how initial oversight was hampered since the more 
than a hundred affected agencies interpreted the Act's access and 
reporting responsibilities differently. A committee print describing 
these and other difficulties was published. The Committee then expanded 
its oversight by administering a detailed questionnaire to agencies 
to determine the ways in which the agencies complied with the law 
during the period 1967 to 1971. After obtaining assistance from the 
CRS in analyzing the responses, the Committee held a set of hearings 
and then prepared a report on problems with administration of the act. 

Recommendations for improvement were included in bipartisan amending 
bills introduced in the House. Simultaneous oversight hearings were 
conducted in the Senate, followed by House and Senate passage of a 
compromise version of a bill to strengthen the Act. The Administration 
opposed the bill and the President vetoed the act. However, the Congress 
overrode the veto on the strength of oversight information generated and 
because the climate — the era of the Watergate scandals — provided 
enough incentive for a bipartisan congressional reaction. 



- J - TG - 12 



168 



Mr. Phillips made two additional important points in concluding his 
remarks. First, he made a clear differentiation, based on his experiences, 
among three different types of operational congressional oversight: 
(1) "routine": that is, monitoring the issuance of agency regulations 
(published in the Federal Register ) to see if the agency is following 
the intent of the law enacted by the Congress; (2) "special": in-depth, 
long-term investigative oversight of a high-priority area which might 
eventually require corrective legislative action, for instance like that 
done on the amendments to the Freedom of Information Act; and (3) "gueril- 
la": spot checking, usually in an adversary or conflict situation where 
a set of one-day hearings prompts corrective action by a Federal agency 
without legislative enactment. 

Second, he noted that it had been exceedingly difficult to do good 
oversight before passage of the Committee Reform Amendments of 1974, 
since most available resources had to be used to prepare new legisla- 
tion. As a result the Congress had to rely almost exclusively on over- 
sight reports prepared by Federal agencies. In many cases these reports 
were biased to give a favorable picture of a sometimes unsavory situation. 
Mr. Phillips said that he expects the House committee reorganization 
and new oversight resources and responsibilities to aid the Congress 
significantly in ensuring good government and maintenance of the balance 
of powers. 

C. Oversight and Evaluation in the Joint Economic Committee: The 
Family Assistance Plan 

For three and one-half years, during the period 1971-1974, the Sub- 
committee on Fiscal Policy of the Joint Economic Committee conducted a 
comprehensive assessment of public welfare, resulting in a 24 volume 



169 



series of hearings, reports and background documents. The objective of the 
study was to assess the effectiveness and impact of the Family Assistance 
Plan, FAP, which established a Federal minimum income floor for all families 
with children. The study was initiated by Representative Martha Griffiths, 
Cnairperson of the Subcommittee and a member of the Ways and Means Committee, which 
had oversight responsibilities for the program. Ways and Means Committee 
deliberations on the issue had convinced Mrs. Griffiths that an adequate 
review of FAP required that the program be assessed in the context of all 
income supplement programs, since changes intended in one program are often 
counteracted or duplicated by other income benefit programs. The other sup- 
plemental benefit programs involved were those offering benefits in food, 
health care, day care, housing, social insurance, social security, and unem- 
ployment compensation. 

Dr. Alair Townsend, research director for the Fiscal Policy Subcommittee 
which conducted the study series, discussed the origins of the study, re- 
sources required to conduct investigatory evaluation and oversight of this 
nature, methods to present findings, and appropriate types of products. 

All the income benefit programs were assessed with respect to (1) the 
distribution of benefits, especially in various combinatons, (2) questions 
of adequacy, (3) the impact on incentives with respect to family structure 
and work effort, and (4) administrative efficiency. 

Ms. Townsend summarized several prerequisites for conducting a 
wide-ranging and lengthy study of this nature. They include: (1) adequate 



170 



leadtime and (2) highly trained professional committee staff cap- 
able of close teamwork. This is necessary to permit the close inter- 
facing required between Subcommittee members and investigative staff 
to keep the study on target and to enable the staff to monitor closely 
the work of the internal staff and outside professionals. Support 
agencies which assisted in the research included the Congressional Re- 
search Service, the General Accounting Office, the House Information 
Systems Office, and outside academic researchers working on contract to 
the Subcommittee. 

The study demonstrated a proliferation and fragmentation of pro- 
grams. Responsibilities for implementation and oversight were frag- 
mented among Federal agencies and State and local governments. The 
research also identified policy problems, inadequacies in data collec- 
tion and analysis, and the need for better syntheses of evaluation re- 
search dealing with the9e issues. 

The Subcommittee reports consisted of 24 volumes of background 
studies and four sets of hearings on issues raised. These detailed 
reports were distributed widely to interested committee staff, interested 
members, Federal agency administrators, and academics. However, in order 
to achieve wide dissemination of results among members of Congress, the 
reports were supplemented by nontechnical summaries, including graphs 
and charts . 

Ms. Townsend concluded by noting that although the Joint Economic Committee 
has no legislative authority, bills containing the essence of the findings 



171 



of the study were introduced by Mrs. Griffiths and members of other 
committees with substantive jurisdiction for issues raised. The study 
received wide distribution and, in Ms. Townsend's opinion, has had a 
significant impact on legislative oversight and evaluation. 

D. Oversight and Evaluation in the Senate Committee on Agriculture 
and Forestry 

James W. Giltmier, a professional staff member of the Senate 
Committee on Agriculture and Forestry, presented the concluding case 
study of the afternoon. He described the legislative need for evalua- 
tion information, but the right kind of evaluation information. Using 
his own experiences as well as examples of other evaluation studies, 
Mr. Giltmier indicated that much evaluation is far too complicated for 
legislative decisionmaking purposes. Evaluation methods which use 
systems analysis models and thoroughly reliable and valid data may be 
required by Federal agencies or academics, but they do not always serve 
legislative decisionmakers who want to know whether or not a program works. 

The General Accounting Office often provides important evaluation 
and oversight information. However, Mr. Giltmier noted that some GAO work 
is not sufficiently tailored to evaluation and oversight purposes. This 
occurs either because the demands of preparing reports required by statute 
leave insufficient time for responding to new demands, or because a given 
study inherently requires a long "turnaround time" and entails a lengthy 
data-gathering and analysis period (as well as a seven-step review procedure) , 
or because a study does not evaluate agency or program performance in terms of the 



172 



"rationale of the [members of Congress] who built the program" but 

rather in terms of narrower concepts of auditing or performance efficiency. 

Mr. Giltmier concluded his presentation by describing some current 
Committee evaluations. These include an evaluation of the Farmers Home 
Administration to determine if the agency has sufficient manpower in 
Washington and in the field to implement its responsibilities. Another 
evaluation effort consists of monitoring recently enacted legislation 
on forestry which included congressionally mandated requirements for 
evaluation. The General Accounting Office assisted the committee with 
both evaluations, helping the committee particularly in drafting legisla- 
tive requirements for evaluation for the forestry bill. 



173 



'FOREIGN AFFAIRS OVERSIGHT: ROLE OF THE STAFF SURVEY MISSION," 
JOHN H. SULLIVAN, HOUSE COMMITTEE ON INTERNATIONAL RELATIONS 



It is a pleasure to have been asked to participate in this seminar 
on Legislative Oversight and Program Evaluation. The subject is one which 
turns me on. Next to legislating — which is the life-blood of any 
committee — oversight is the reason we staff people get paid. 

The Committee and Oversight 

The committee I come from — once the Committee on Foreign Affairs 
and now the Committee on International Relations — has a long record 
of concern for oversight and review. 

One of the most written-about Congressional study missions in re- 
cent history was the Javits Mission to Europe early in the 1950 's which 
found abuses in the Marshall Plan programs and recommended a series of 
reforms most of which ultimately were adopted. 

Some 26 years ago the Committee formed a special Subcommittee for the 
Review of Foreign Aid Programs, chaired by the Chairman of the full Commit- 
tee with membership by top ranking members of both parties. Most of the 
staff oversight studies and many of the Congressional study missions were 
undertaken under the auspices of this Review Subcommittee. Its work 
resulted in U.S. foreign operations, especially in the area of foreign 
aid. 

More recently, in this 94th Congress, the Committee revised its sub- 
committee structure in keeping with the Committee Reform Act of 1974 and 
new jurisdiction which was given to it. Right now it has three subcommittees 
with essentially an oversight function. They are: 



174 



The Subcommittee on Oversight , chaired by Chairman Morgan. It deals 
with oversight of foreign intelligence; administration of foreign aid; 
external research — the General Accounting Office, Congressional Research 
Service; Inspector General of Foreign Assistance; Auditor General of AID; 
and all areas not covered by other subcommittees. 

The Subcommittee on Investigations , chaired by Congressman Lee Hamilton 
of Indiana, which is to deal with special foreign policy-related investiga- 
tions referred by the chairman, with the consent of the committee. Most 
recently that subcommittee has been studying the entire question of United 
States policies in Indochina. It has played a constructive role in keeping 
the committee abreast of the fast-moving events in Indochina. 

The Subcommittee on Future Foreign Policy Research and Development . This 
subcommittee was formed pursuant to the mandate in the Committee Reform Act 
of 1974 that committees undertake "futures research and forecasting". To my 
mind that encompasses a large measure of oversight. One cannot go about 
prescribing for the future until one understands the mistakes and problems 
of the past. 

Yet another manifestation of the seriousness with which the Chairman 
and members of the Committee on International Relations^approach the over- 
sight function is the biannual oversight report required by the Legislative 
Reorganization Act of 1970. 

Some committees seemingly have undertaken those reports as if they were 
a nuisance. Others have included oversight information in their "surveys of 
activities" — thus lumping together their legislative and oversight activities. 

The Committee on International Relations — like the Ways and Means 
Committee and the Education and Labor Committee — have seen the biannual 
report requirement as an opportunity to pull together in one place a 
comprehensive account of oversight activities, 



175 



It has been my hope that the House Committee on Government Operations 
would begin looking at those reports and, in effect, "giving grades". 

Oversight of Overseas Operations 

My purpose here today is not to talk about the Committee, but about means 

of conducting Congressional oversight on U.S. operations overseas. 

Basically, there are three ways such oversight has been carried out: 

First, through Congressional study missions in which members of Congress — 
a delegation, several, or sometimes just one — go abroad to look one or several 
foreign affairs problems. 

Second, staff survey missions in which professional staff members of 
relevant committees are sent abroad by their Committees to obtain information 
on problems and to report back to the members. 

Third, the General Accounting Office often carries out studies 
which involve their personnel abroad — or personnel from the Washington 
office who go abroad — in order to obtain information for reports requested 
by Congressional committees. 

Occasionally Congressional Research Service staffers may become involved 
in such studies. Several years ago two staff members here went to Europe to 
gather information on the continuing usefulness of Radio Free Europe and 
Radio Liberty — a study which attracted considerable attention when it was 
released. 

My purpose today is not to talk about Congressional study missions 
in which the Members, themselves participate. Not because I doubt their 
ability to uncover important information affecting policy. You might recall 
that it was just such a Congressional delegation visiting Vietnam in 1960 which 
uncovered the infamous "tiger cages" at Con Son Island. I exclude such 
missions because their character so often if diffuse. 



176 



Some are serious fact-finding efforts and some are not. Just as the 
American press, unfortunately, is unable to tell the difference, discussion 
of them here today might lead far afield from our focus on oversight activi- 
ties. 

Nor do I wish to speak about GAO and other activities overseas. They 
provide a valuable back-up to us in Congress but they do not have — and 
possibly will never have — the "inside" impact which a staff report can 
have. 

Before launching into some case studies and examples of such staff 
studies — both from the Committee on International Relations and the 
Senate Committee on Foreign Relations — let me make several preliminary 
points: 

First, the information which can be obtained through a staff survey 
mission often can be obtained in no other way. 

Second, the professional qualifications and attitudes of those staff 
members undertaking the study are essential to their success. 

The staff members must have the trust — or at least the respect — 
of the officials of the executive branch and others with whom they deal. More 
important, they must have the trust and respect of the members of Congress. 

Having this trust, they then should be free to exercise their own inde- 
pendent judgments of situations — whether or not their findings accord with 
the prevailing wisdom of their committee or the executive branch. 

Third, information uncovered by a Congressional staff person and con- 
veyed to influential members of Congress can be a powerful force for changing 
the executive branch policies which are at conflict with the mandates of 
Congress. 



177 



Moose and Lowenstein, Moose and Meissner 

By far the most influential staff survey team has been Moose and 
Lowenstein, Dick Moose of the Senate Foreign Relations Staff and Jim 
Lowenstein, now back in the Department of State. Dick Moose now has 
a new partner — Chuck Meissner, formerly of Sen. Percy's staff, now 
on the Foreign Relations Committee staff. 

This new duo seems equally able to afflict the bureaucratic estab- 
lishment. 

It was Moose and Lowenstein who in 1973 discovered that the United 
States bombing in Cambodia was being coordinated from the United States 
Embassy in Phnom Penh. That revelation burst like a B-52 blockbuster in 
Congress. It looked to many that Cambodia was a new tunnel at the end 
of the light, with growing U.S. involvement. Thus, two months later 
the Senate passed the Church-Case amendment to halt the bombing in that 
country. 

It was to Moose and Lowenstein in the fall of 1973 that President 
I Marcos revealed for the first time that an assassination plot against 
1 him has been uncovered — a plot which included the hiring of American 
gunmen to do the killing. This information has probably resulted in 
somewhat more sympathy f° r Marco's martial law decrees on Capitol Hill 
than otherwise might have been the case. 

Most recently, it was Moose and Meissner who found out that the 
United States Ambassador in Saigon, Graham Martin, was dragging his 
feet on the evacuation of Americans from Vietnam when it seemed in- 
evitable that the country would fall to the Communist advance. They 



178 



found that planes leaving the country for safe havens were going with 
many seats empty — though many persons were clamoring to leave. 

This information caused senators to put pressure on the President to 
give first priority to getting the Americans out, which he ulitmately did. 
American numbers had been reduced to around one thousand, four hundred, by 
the time the final helicopter lift occurred. 

But it is much too tempting to emphasize the dramatic aspects of 
these Senate staff missions. Beyond these headline making aspects of 
their investigations is a great deal of solid detail — much of it at 
variance with the official wisdom. As the Washington Monthly recently said, 
the cables and reports sent back by these Senate investigators — and 
I quote — "provided the Senators with a steady stream of fresh, undiluted 
information." 

House Staff Survey Missions 

Although the staff survey missions undertaken by the House Foreign 
Affairs Committee have not received the same public attention as those 
of their Senate counter-parts, they had their moments. 

For example, in the fall of 1972, Jack Brady and Bob Boyer of our 
staff spent three weeks investigating the narcotics trade in Southeast Asia. 
By small plane they flew into the no-man's-land known as the "Golden 
Triangle" where Burma, Laos and Thailand come together. 

The area is notorious for drug smuggling. 

While there Brady and Boyer uncovered what has come to be known as the 
"extra ton of opium". 

The U.S. Bureau of Narcotics and Dangerous Drugs had agreed with 
the Thai Government to what amounted to a pre-emptive buy of opium from the 



179 



Chinese Irregular Forces who have been occupying northern Thailand for 
years and funding their activities with opium-growing. 

The U.S. turned over $1 million dollars. When the Chinese Irregulars 
showed up at the turn-in point they had 27 tons of opium rather than 
the 26 which had been agreed upon. 

Thai and U.S. authorities refused to accept the extra ton and the 
Chinese were ordered instead to get it out of Thailand. 

Our men were highly critical of that action — the expectation 
being that the opium probably found its way to U.S. troops who then were 
still in Vietnam. 



On a 1972 mission looking into U.S. -funded family planning programs 
in Asia, my partner — Chips Chester — and I uncovered the fact that sitting 
in a Stockton, California warehouse were 625 vasectomy/lUD surgical kits 
valued at $625 thousands. 

They once had been destined for Korea. But there was no way for the 
Koreans to use them. So the Agency for International Development was 
storing them, not knowing what to do with them. 

Time does not permit an examination of that particular bureaucratic 
blunder. Suffice to say that it resulted from the actions of an official 
with too much foreign aid money to spend, and too little wisdom on how it 
should be expended. 



A much more important finding from that mission was that the birth 
control pills which AID was distributing to women in Asia were having 
adverse physical reactions and the program was suffering thereby. 



180 



The problem arose because the kind of pill distributed each year depended 
on which American drug company has the lowest bill. Each year for five years 
a different company was awarded the contract. 

As a result a woman in the Phillipines or Indonesia who had been in the 
program for 5 years was switched to a new pill every year. In 1972, the switch- 
over produced a very worrisome side effect — and women were dropping out of 
the birth control program in droves. 

Dr. Ravenholt who runs AID's Population Bureau believes the pill has no 
adverse side effects — in fact, is good for the health. 

He wasn't too interested in hearing about these problems and his population 
officers in the various countries were not too keen on reporting the bad news to 
the boss. 

But we heard about it from the local doctors and some of the women who were 
taking the pill. 

We described the problem in our report — at which point Dr. Ravenholt sent 
three doctors to Asia to check out the situation. Their report confirmed our 
findings. 

As a result, AID changed its pill distribution practices to minimize future 
such problems. 

Incidentially, after reading that report one cdlumnist was moved to call 
Chips and me the "Masters and Johnson of Capitol Hill." 

Head Count in Cambodia 

The case study which I want to discuss in some detail has to do with the 
effort by the congress to minimize United States involvement in Cambodia. 

In 1972, the members of the Senate Foreign Relations Committee became 
alarmed at what appeared to be growing United States involvement in Cambodia. 



181 



As a means of dealing with the issue, the Committee appended an amendment 
to the Foreign Assistance Act of 1972 setting a ceiling of 200 on the number 
of American officials who could be in the country, and a ceiling of 75 third 
country nationals working on contract with our mission, aiding the Cambodian 
armed forces. 

Since the matter would be a point at conference, the Committee in the 
fall of that year sent Bob Boyer and me to Cambodia to assess the entire 
situation and to recommend whether the ceiling numbers should be accepted 
and at what level. 

In the course of our investigation we talked to dozens of U.S. and 
Cambodian officials about the numbers. We even took our own head count to 
find out how many official Americans were in country at that time. On our 
own we walked into the Air American office to count. We visited a semi- 
secret communications facility. And we added numbers to numbers. 

When we were finished we cabled our members that a 200 ceiling seemed 
like a reasonable figure. While we were still abroad the House conferees 
yielded to the Senate on that point. 

In the spring of 1974, Jack Brady and I visited Cambodia as part of 
a staff study of military and economic aid in Indochina. 

In the meantime the situation in the country had deteriorated and the 
U.S. mission had become considerably more activist in its relations with the 
Cambodians. 

The 200-person ceiling had begun to pinch. 

As a result, despite the plain wording or tne law and its obvious intent, 
we found that an average of 215 to 222 American civilian and military officials 
were daily at work in the U.S. Mission in Phnom Penh or elsewhere in the country. 

The rationale provided by the Embassy was that not more than 200 American 
officials ever were present in Cambodia overnight . They admitted that more 



182 



than that number were present in Cambodia at any one time during working 
hours. 

In order to insure that no more than 200 official Americans were present 
overnight, the Embassy had initiated a procedure they called "headspace 1 . 1 . 

Under "headspace' the Deputy Chief of Mission saw to it that sufficient 
personnel were out of the country by nightfall to accomodate those on TDY who 
might be staying overnight. 

In many cases individuals deemed expendable traveled to Bangkok for 
several days of sunning by the pool, all the time being carried on the work 
rolls and collecting per diem for the vacation. 

This procedure was a clear attempt to circumvent the law — and an 
expensive one at that. The government was forced to pick up air fares both 
for those going our and those coming in, plus salary for- days not worked, 
plus per diem. 

When we confronted the Ambassador with this scheme, he denied that it 
violated the law. We let him know that our opinion was otherwise. 

While we were in the process of writing our report after returning to 
Washington, we received information that the Embassy was reducing its daily 
work force back to 200. 

That information was contained in our report to the Committee. But be- 
cause it was not corroborated, we suggested that Congressional action might 
still be needed to remedy the clear abuse of the law. 

Although no legislative action was taken, several congressmen filed a 
legal suit against the Administration and various federal officials, citing our 
report, and charging them with violations of law. 

I should note that our initial information about the abuse came from the 
GAO regional director, Fred Lyons, with whom we regularly meet when we are in 
that area. But Fred had only some of the facts: he was blocked from getting the 



183 



rest because of limited access to the people and documents involved. We had 
the access he lacked and could complete the picture. 

Knowing how bureaucratic processes tend to take on a life of their own, 
the "headspace practice" might well have expanded even beyond the 15 or so 
extra people daily — especially as the Cambodian situation deteriorated 
further. 

That would have been just that many more to have had to evacuate when the 
final fall of Phnom Penh occurred. 

Not long before that fall — in February of this year — Jack Brady and I 
made another visit to Cambodia, as staff members assigned to the Congressional 
delegation which went to Indochina at the behest of the President. 

The Ambassador was not happy to see us. He has been named a defendant in 
the legal case. But he confirmed that the "headspace" practice had come to an 
end soon after our last visit. 

This incident points up some of the points I made earlier. 

Often the information can be gotten no other way except through a fact- 
finding mission abroad. In this instance, though someone in Washington obviously 
knew about the headspace violation, the information was not of the type to be 
volunteered to Congress. It had to be found out in Cambodia. 

Second, the experience and background of the investigators is important. 
I had been in Cambodia before and knew the law on the 200-man ceiling. Jack 
Brady has a U.S. Army background which gave him an understanding that the 
personnel "bookeeping" system involved in "headspace" was an adaptation of a 
military procedure. 

Third, the uncovering of information of this sort can result in changes 
of Executive Branch policies which are at variance with Congressional intentions. 



184 



Two Additional Points 



Before closing, two additional points deserve discussion. 

First, staff study trips are best taken in pairs, or with three persons. 
Solo performances should be left to the members. 

Why? Because two heads being better than one is not just a musty adage — 
it is true. Second, for corroboration of evidence. Executive Branch officials 
sometimes have a tendency later to deny what they have told a Congressional 
investigator. Rather than have one man's word against another, it is far 
preferable to have one man's word against two others. 

On the Committee on International Relations we must, by Committee practice, 
travel in pairs. 

In the Senate, however, some members of the Foreign Relations Committee 
sometimes found it hard to understand why both Moose and Lowensteln had to 
go, when a single air fare might have sufficed. 

Second, as a result of the Moose and Lowenstein successes, some politi- 
cal scientists have suggested that the Congress establish watchdog staffs in 
various parts of the world to provide the members with information. 

That just won't work. Staff members who are far removed from the 
Committee and the Congress they are to serve will lose the daily close con- 
tact with members which is essential to building trust and rapport. Such 
staffs would be able to do little more than the GAO now is able to do. 

At the same time, it might be advisable for the relevant committees to 
increase the number of teams available for investigations abroad. The 
Committee on International Relations has acted recently to strengthen its 
capabilities by hiring a young auditor and attorney with experience in re- 
viewing foreign assistance programs abroad. 



185 



Conclusion 

In conclusion, It must be remembered that the Congressional staff survey 
abroad is only one of a number of ways in which the Congress can exercise its 
oversight responsibilities in the area of foreign affairs. 

This oversight process must be a continuous one, involving hearings, 
studies and other forms of research into executive branch activities abroad. 

But the staff study has proved its worth and is on its way to becoming an 
institutionalized method of providing necessary oversight information to the 
Congress. 



186 



DISCUSSION 

Several points especially pertinent to the relationship between 
evaluation and oversight were raised in the discussion following Dr. 
Sullivan's paper. 

As noted by a previous speaker, there is a difference between 
process and impact evaluation. Process evaluations answer questions 
about the efficiency of program management and use data normally col- 
lected in administering a program. Impact evaluations are designed 
to answer questions about whether a program caused a change in the 
participants' behavior and or attitudes. Data usually collected in 
the normal course of administering a program are insufficient to an- 
swer impact questions. The differnce between these two types of eval- 
uations was demonstrated in the committee's investigation of the use 
of birth control pills in Southeast Asia. The Agency for Internation- 
al Development collected information on the distribution of the pills 
and correlated this with birthrate data. However this type of pro- 
cess evaluation was insufficient since it did not provide informa- 
tion to describe the side-effects of birth control pills on the women 
participating in the program. Thus in this case, the process evalu- 
ation criterion of efficiency, measured in terms of distribution of 
birth control pills was not sufficient to identify problems relating 
to the adverse impacts of the program. 

It was also noted that since agencies have vested interests in 
maintaining programs and budget, frequently they are reluctant to 
report problems which might indicate program failure or deficiency. 



187 



Several questions were raised about the pros and cons of using 
GAO versus committee staff investigators in overseas oversight ac- 
tivities. It was noted that GAO investigations sometimes are in- 
adequate. Several reasons were given. First, agency field per- 
sonnel may accord more "weight" or status to committee investiga- 
tive staff than to GAO staff investigators. Also GAO investigations 
usually take longer to do. However, frequently GAO investigative 
staff develop important leads, which committee staff then follow. 

Several additional remarks were made about the use of commit- 
tee investigative staff for overseas field work. Dr. Sullivan re- 
marked that usually it is advantageous to detail two or three staff 
to an assignment, since it may be necessary to have someone cor- 
roborate information which may later be denied. It may also be 
useful to use both majority and minority staff in such investiga- 
tions since, unlike GAO staff, committee staff members have rap- 
port with committee chairmen and members and immediate access to 
them. 



188 



"THE FREEDOM OF INFORMATION ACT," WILLIAM G. PHILLIPS, FORMER STAFF 
DIRECTOR (1971-1975) , FOREIGN OPERATIONS AND GOVERNMENT INFORMATION 
SUBCOMMITTEE, HOUSE COMMITTEE ON GOVERNMENT OPERATIONS 22_/ 



I. Introduction 



This case study of legislative oversight by the House Foreign 
Operations and Government Information Subcommittee 40 / of the Govern- 
ment Operations Committee represents a unique situation in several 
respects. First, it involves a subcommittee's activities where the 
same Government Operations Committee unit has both oversight and 
legislative jurisdiction, and is one of the few cases where the Gov- 
ernment Operations Committee exercises such broad substantive leg- 
islative authority. Moreover, the case study, which spans a period 
of 3 1/2 years, began with an intensive series of oversight inves- 
tigations and hearings on the administration of the Freedom of Infor- 
mation Act (5 U.S.C. 552), and proceeded through every step of the 
legislative process as well — hearings on recommended amendments, 
committee action, floor action and passage in the House and Senate, 
and finally culminating in their enactment into law over a Presi- 
dential veto. 



39 / For a detailed legislative history of the 1974 amendments to the 
Freedom of Information Act, see Joint Committee Print , House Gov- 
ernment Operations Committee and Senate Judiciary Committee, "Free- 
dom of Information Act and Amendments of 1974 (P.L. 93-502)." 
Washington: U.S. Government Printing Office, March, 1975. 

40 / The name of the subcommittee was changed to "Government Informa- 
tion and Individual Rights" at the beginning of the 94th Congress 
(January 1975) . 



189 



The Freedom of Information Act of 1966 (FOIA) (P.L. 89-487) ^jj 
was a milestone enactment, providing that "any person" might have ac- 
cess (with certain exceptions) to rules, opinions, orders, and other 
types of public records of various agencies of the executive branch of 
the Federal Government. The product of more than a decade of investiga- 
tions, studies, reports, and vigorous advocacy for the "people's right 
to know," the 1966 law was an amendment to section 3 of the Administra- 
tive Procedure Act of 1946. It reversed long-standing information pol- 
icies and practices of the Federal executive bureaucracy by establish- 
ing the right of public access (including corporate entities) to various 
types of identifiable Federal agency records not falling within the 
nine categories of exemptions specified in the act. It gave the public 
the right to sue in Federal court to obtain the records. There was 
no requirement that a person be required to state his reasons for want- 
ing the information and it further required that the burden of proving 
the legal justification for withholding of requested records in each 
case be the responsibility of the Federal agency involved. 

Withholding of information by a Federal agency must be justified 
on the basis of one or more of the nine specific exemptions permitted 
by section 552(b) of the act. They relate to matters that are: 

(1) Specifically required by Executive order to be kept secret 
in the interest of the national defense or foreign policy; 

(2) Related solely to the internal personnel rules and practices 
of an agency; 



41/ Codified as Section 552, title 5, U.S. Code by P.L. 90-23, with 
slight modifications, on June 5, 1967. 



190 



(3) Specifically exempted from disclosure by statute; 

(4) Trade secrets and commercial or financial information ob- 
tained from a person and privileged or confidential; 

(5) Inter-agency or intra-agency memorandums or letters which 
would not be available by law to a party other than an agency 
in litigation with the agency; 

(6) Personnel and medical files and similar files the dis- 
closure of which would constitute a clearly unwarranted in- 
vasion of personal privacy; 

(7) Investigatory files compiled for law enforcement purposes 
except to the extent available by law to a party other than an 
agency; 

(8) Contained in or related to examination, operating, or con- 
dition report prepared by, on behalf of, or for the use of an 
agency responsible for the regulation or supervision of finan- 
cial institutions; or 

(9) Geological and geophysical information and data, including 
maps, concerning wells. 

The act makes it clear in section 552(c) that the exemptions have 

absolutely no effect upon congressional access to information: 

This section does not authorize withholding of information or 
limit the availability or records to the public, except as spe- 
cifically stated in this section. This section is not authority 
to withhold information from Congress. 

Investigative efforts on information policies and practices of 
Federal agencies were initiated in 1955 by the subcommittee's predeces- 
sor — the Special Subcommittee on Government Information. They were 
spearheaded in the House by Representative John E. Moss (D., Calif.), 
its chairman until 1971. Similar activities were carried on in the 
Senate by the Administrative Practice and Procedure Subcommittee of 
the Judiciary Committee, first under the chairmanship of Senator Thomas 
Hennings (D., Mo.), and later by Senator Edward V. Long (D., Mo.). 

Voluminous House and Senate hearings, staff studies, and commit- 
tee reports disclosed the widespread "secrecy practices" employed by 



191 



the executive bureaucracy. Increasing attention was also being di- 
rected at abuses of Government secrecy by the news media, its pro- 
fessional organizations, and distinguished students of journalism 
such as the late Harold Cross. Under the guise of "national security 1 
or the cloak of "administrative anonymity," vast quantities of infor- 
mation generated by Federal agencies were hidden from public scru- 
tiny, often to avoid political embarrassment, bureaucratic mistakes, 
administrative decisions contrary to the public interest, cost over- 
runs, or to cover up conflicts of interest, or corruption. 

When congressional hearings were held in the Johnson adminis- 
tration on legislation to correct these abuses, executive branch 
witnesses were unanimous in their opposition. Freedom of Informa- 
tion legislation, they contended, would destroy the effective op- 
eration of Federal agencies by bogging them down in processing vast 
numbers of requests for information from citizens and the news me- 
dia. Even after Congress had unanimously passed the bi-partisan 
bill which subsequently became the Freedom of Information Act in 
1966, secrecy-minded bureaucrats pleaded with President Lyndon 
Johnson to veto the measure. But Johnson ignored these pleadings 
and signed the Freedom of Information bill into law on July 4, 
1966, stating that "a democracy works best when the people have 
all the information that the security of the Nation permits." It 
became effective one year later — July 4, 1967. 

Another unique aspect of this milestone law is that each Fed- 
eral agency — presently more than 100 — administers the provisions 
of the FOIA within the agency, based on its own agency regulations 
promulgated in the Federal Register and based on legal interpretative 



192 



guidelines of the Attorney General. Only minimum consultative as- 
sistance is provided to agencies by the Justice Department's Office 
of Legal Counsel through its Freedom of Information Committee, es- 
tablished as an informal information clearinghouse in December, 1969. 
This means that congressional oversight over the administration of 
the FOIA must necessarily involve scrutiny of the information ac- 
tivities of each of the more than 100 Federal agencies subject to 
its provisions — a most tedious, time-consuming task for the hand- 
ful of House and Senate staff personnel assigned the task by the 
two subcommittees having jurisdiction over the act. 

II. House Oversight Plan Formulated in 1971 

During the four years following the effective date of the FOIA, 
House and Senate subcommittees had engaged in only sporadic, limited 
types of oversight activity. Initial agency regulations to implement 
the act were reviewed for compliance with congressional intent and a 
compilation was published as a House Committee Print . The Senate com- 
mittee had compiled a 10 month review of the operation of the new law. 
Both staffs had on numerous occasions acted as "informal ombudsmen" 
in providing information and assistance to congressional offices, his- 
torians, the news media, and the general public involving requests to 
agencies for Government information under the FOIA. 

The reorganization of the House Government Operations Committee 
at the beginning of the 92nd Congress resulted in the appointment of 
a new subcommittee chairman, Representative William S. Moorhead (D., 
Pa.). One of the priority targets agreed upon by the subcommittee was 
a thorough investigation of the operation of the four year old FOIA, 



193 



which had then operated for approximately two years under a Democratic 
(Johnson) administration and two years under a Republican (Nixon) ad- 
ministration. 

The FOIA oversight plan, as formulated in the spring of 1971, in- 
volved several stages to take full advantage of dramatic events that 
focused on the latest public disclosures of widespread Government se- 
crecy. Late in June, the subcommittee commenced public hearings on the 
various issues of Government information policies and withholding prac- 
tices involved in the so-called "Pentagon Papers" controversy. These 
hearings and follow-up subcommittee inquiries in 1972 focused on clas- 
sification operations within the executive branch under the security 
classification system established by President Eisenhower in 1953 (Ex- 
ecutive Order 10501) . These hearings also explored the relationship 
of such "classified" types of information to the FOIA exemption, legal 
and constitutional questions involved in publication of the "Pentagon 
Papers" by the New York Times and Washington Post , and matters of pub- 
lic policy in the field of Government information. 

The next stage was the extensive preparation for in-depth over- 
sight hearings by the subcommittee on the administration of the FOIA 
by Federal agencies. A detailed questionnaire was formulated by mid- 
summer 1971 and was sent to each of the Federal agencies subject to 
the FOIA. The questionnaire sought voluminous types of statistical 
records from the agencies — 

* How many formal requests were made for information under the 
act? 

* How many requests were granted? How many denied? Which of 
the nine exemptions was involved in each denial? How many 
were granted after intra-agency appeal processes? How many 
such cases ended up in the courts? What was the outcome in 
each case? 



194 



* What types of groups, corporations, or individuals made re- 
quests for information under the FOIA? How many requests 
for each category of requester? 

* How does the security classification system of the agency func- 
tion under Executive Order 10501? How many officials are au- 
thorized to classify information under each of the three clas- 
sification levels? 

The data requested was to cover the period of the effective date of 
the FOIA (July 4, 1967) through July 4, 1971. Before the question- 
naire was formally sent to executive branch agencies, it was "pre- 
tested" by discussing possible questions with a number of Government 
officials who would have the eventual responsibility of responding 
to the questionnaire. Because of the volume of data requested, a 3- 
month period was provided for agency replies . All but a few agen- 
cies complied, although several claimed they had kept no such rec- 
ords on the FOIA. 

The expert assistance of the Congressional Research Service, 
Library of Congress, was solicited at an early stage in the for- 
mulation of the oversight plan. Responses to the questionnaire were 
analyzed and tabulated by Dr. Harold Relyea and Ms. Sharon Gressle 
of the Government and General Research Division of CRS. Other re- 
lated research papers, case analyses, and background data were also 
prepared. Additional expert consultant assistance in this project 
was also provided by CRS to the subcommittee staff and Members. 

The final- analyses and tabulations of data provided by agencies 
in response to the subcommittee questionnaire were completed by CRS 
personnel early in 1972 and preparations for oversight hearings in 
the spring began. A group of representative executive agencies were 
selected to appear at the hearings on the basis of the questionnaire 



195 



analysis. They included old-line departments, newer departments, in- 
dependent regulatory agencies, boards, military services, executive 
agencies, commissions — a broad cross-section of the executive bu- 
reaucracy. 

Those with better than average records of compliance with con- 
gressional intent in the administration of the FOIA were included, 
along with other agencies whose records of administering the FOIA 
were a negative quantity. Subcommittee staff personnel spent sev- 
eral weeks in individual conferences with the key officials of de- 
partments and agencies to be called to testify at the oversight 
hearings. Additional information to supplement questionnaire data 
was obtained in this way. Information provided by the agencies 
was also clarified in some cases; dates of appearance and names of 
the most knowledgeable agency officials to testify were likewise 
negotiated during this informal staff consultative process. Exten- 
sive efforts were also made by the subcommittee staff to solicit 
the testimony of certain key non-governmental witnesses who could 
provide expert testimony from their vantage point on the positive 
and negative aspects of agencies' administration of the FOIA. 

The subcommittee's public oversight hearings on the administra- 
tion of the act consumed fourteen days of intensive inquiry during 
the months of March and April, 1972. Some fifty witnesses testified, 
including spokesmen for a broad cross-section of executive depart- 
ments and agencies, plus news media representatives, attorneys ex- 
perienced in FOIA litigation, former Government officials involved 
in FOIA matters, academicians, and representatives of organizations 
utilizing the FOIA. 



196 



The next stage of the subcommittee's oversight work took place 
during the months of May and June, 19 72. Additional hearings were 
held on special aspects of the FOIA's operations, including several 
days on the functioning of the security classification system in the 
executive branch under Executive Order 10501 and the ways in which 
that system would be affected by the new Executive Order 11652, is- 
sued by President Nixon in March, 1972. Other hearings were held on 
the issues involving the right of Congress to information from the 
executive branch and on the information functions of the approximately 
2,000 Federal advisory committees which exert tremendous influence on 
governmental policy decisions — mostly in meetings held behind closed 
doors. 

All told, the 1971-1972 oversight hearings of the subcommittee 
— spanning a 12-month period — involved 41 days of public hearing; 
testimony was received from 142 witnesses who furnished 9 volumes of 
valuable information on every conceivable aspect of operations of the 
FOIA and matters relating to the dissemination, as well as the with- 
holding, of information by executive agencies of the Federal Govern- 
ment. 

With the conclusion of the subcommittee's series of FOIA over- 
sight hearings in June, 1972, work began immediately on the next 
stage of the plan — the drafting and consideration of a comprehen- 
sive investigative report based on these hearings. The initial staff 
draft was completed during the summer, edited and revised during bi- 
partisan staff meetings, and circulated among members of the subcom- 
mittee. It was further revised during subcommittee mark-up sessions, 
unanimously approved, and forwarded to the full Government Operations 



197 



Committee membership for further consideration. There it was discussed, 
further amended, and unanimously adopted by the full committee late in 
September, 1972. The 90-page report was subsequently issued as H. Rept. 
92-1419. ("Administration of the Freedom of Information Act"). 

The report described the major problem areas in the operation of 
the act, documenting each by specific examples from the hearing record. 
They included: (1) bureaucratic delays in responding to requests for 
information; (2) abuses of fee schedules for search and copying of re- 
quested records; (3) cumbersome and costly legal remedies to contest 
agency withholding of requested information; (4) the lack of involve- 
ment of public information personnel within agencies in the decision- 
making process affecting disclosure or withholding of requested records; 
(5) the relative lack of use of FOIA by the news media; and (6) the 
lack of a more positive attitude toward FOIA by top-level administra- 
tors of executive agencies and departments. 

The report also made a series of recommendations to the agencies 
to improve their administration of the act and set forth a separate 
series of recommendations calling for substantive amendments to the 
FOIA requiring further legislative consideration by the committee. Bi- 
partisan bills were introduced by Chairman Moorhead and other members 
to implement these legislative recommendations during the next several 
months. This led into the next stage of the subcommittee's plan — 
legislative hearings on proposed amendments to strengthen and improve 
the operation of the FOIA. 

Five days of legislative hearings were held on such bills by the 
subcommittee during the first two weeks in May, 1973. Executive branch 
witnesses flatly opposed any amendments to strengthen the FOIA. However, 



198 



the legislation was supported by the American Bar Association's Ad- 
ministrative Law Section, news media witnesses, and many other or- 
ganizations and individuals familiar with bureaucratic subversion of 
congressional intent of the original FOIA. 

Subcommittee mark-up sessions continued through the fall in an 
effort to obtain a bi-partisan consensus on a package of workable 
amendments to the act but final agreement was not reached until early 
the following year. A clean bill representing the bi-partisan com- 
promises reached during the subcommittee mark-up meetings was introduced 
in January, 1974 and the measure was cleared for action by the full 
Government Operations Committee. 

The bill (H.R. 12471) was considered by the full committee and 
unanimously reported to the House on February 21, 1974. (H. Rept. 93- 
876). It contained more than a dozen substantive amendments to the 
act to correct the major deficiencies in its operation described in 
the committee's investigative report. A few weeks later the bill was 
debated on the House floor and passed, with minor technical amendments, 
by a vote of 383-8 and sent to the Senate. 

Meanwhile, during the spring of 19 73, the Administrative Practice 
and Procedure Subcommittee of the Senate Judiciary Committee, chaired by 
Senator Edward M. Kennedy (D., Mass.), and two Senate Government Oper- 
ations Committee subcommittees having overlapping jurisdiction in the 
field of the FOIA and Government secrecy generally had also held exten- 
sive joint hearings in this same area. Some 11 days of hearings were 
held and testimony was received from over 40 witnesses, amounting to 3 
volumes of printed proceedings. The Senate version of the 1974 FOIA 



199 



amendments, quite similar to the House bill, was favorably reported 
early in May and passed the Senate, with amendments by a vote of 64- 
17 on May 30th. 

The House-Senate conference committee met in public sessions on 
4 days during the month of August, 1974 to resolve differences between 
the two bills. The meetings overlapped the resignation of President 
Nixon and the swearing-in of Vice President Ford as President. The new 
President requested that the conferees defer further conference action 
until he had an opportunity to study the legislation and to submit his 
views. The request was honored and President Ford subsequently trans- 
mitted a letter to the conference committee detailing several major ob- 
jections to specific provisions of the measure. The conferees later 
agreed to several important compromises in an effort to reach an ac- 
commodation of conflicting executive and legislative views. Conference 
agreement was finally reached in September, 1974 and the conference re- 
port was routinely adopted early in October by both the House and Senate 
and the bill was sent to the White House. 

Just prior to the congressional recess for the November elections, 
President Ford surprisingly vetoed the legislation. Virtually every 
executive agency had recommended such action, just as the bureaucracy 
had recommended to President Johnson that he veto the original 1966 
law. But the Ford veto, on the heels of his public pledge to advance 
the cause of "open Government" was unexpected. The dramatic unfolding 
of the sordid details of the Watergate coverup, paralleling congres- 
sional consideration of the FOIA amendments in 1973 and 1974, had ob- 
viously helped to persuade Congress and the public of the need for a 
stronger law to protect the public's "right to know". 



71-867 O - 7fi - 14 



200 



The Presidential veto of the 19 74 FOIA amendments was considered 
by the House on November 20th and overriden by a vote of 371-31. The 
Senate followed suit the next day by a 65-27 vote and the bill became 
P.L. 93-502. The amendments became effective on February 19, 1975 — 
90 days after enactment. 42/ 

III. Some General Observations 

General observations below on the role of a congressional subcom- 
mittee or committee in its exercise of positive oversight and/or pro- 
gram evaluation are based on both the case history of the 1974 amend- 
ments to the Freedom of Information Act as set forth above and on 
extensive work of the Foreign Operations and Government Information 
Subcommittee in the foreign operations area of its jurisdiction during 
the same 1971-1975 period. 

Much of the subcommittee's activity in this part of its jurisdic- 
tion was concentrated on the operation of various types of U.S. assis- 
tance to South Vietnam, Laos, and Cambodia in the late 1960's and early 
1970' s through programs administered by the Agency for International 
Development (AID) . 

It held extensive investigative hearings into such subject areas 
as the operation of the illegal black-market in these countries, nar- 
cotics trading, the smuggling of gold, diversion of U.S. funds and 
property, the so-called "Phoenix" program, refugee relief, health pro- 
grams, the U.S. propaganda assistance provided the Government of South 



42 / Details of the 17 major substantive amendments to the 1966 act in- 
corporated in P.L. 93-502 are discussed at pages 113-116 of the 
Joint Committee Print (March, 1975) cited above. 



201 



Vietnam (GVN) by the USIA, the conduct of the South Vietnamese na- 
tional elections, public works assistance, currency stabilization 
programs, and many other non-military assistance efforts provided 
by the U.S. to the then existing Governments of these three coun- 
tries. 

During these investigations, the subcommittee utilized a num- 
ber of outside investigative sources. They included field work by 
General Accounting Office personnel in Vietnam on the "pacification" 
program that resulted in a report and subsequent testimony at sub- 
committee hearings on the subject by top officials of the GAO's 
International Division. Other information was obtained through 
cooperative agreements at the staff level with personnel of the 
House Foreign Affairs and Senate Foreign Relations Committee. On 
a number of occasions, valuable leads for investigative follow-up 
were obtained from "whistle-blowers" with first-hand information 
about what they felt were violations of congressional intent or 
outright corruption in the administration of various AID programs 
in these countries. Other useful data was obtained by subcommittee 
staff investigators through interrogations of agency personnel or 
through information furnished by AID witnesses called to testify 
on specific allegations or on general program activity being con- 
ducted in those countries. 

Throughout the foreign operations portion of the subcommittee's 
oversight functions during this period, there was an "adversary" re- 
lationship with the executive agency. Members and staff became wary 
of the value of official information furnished by AID officials be- 
cause of the many examples of inaccurate or misleading data provided 



202 



during hearings or on request to the agency. During one of the sub- 
committee's investigation of the AID program in Cambodia, President 
Nixon exercised "Executive privilege" to withhold documents that had 
been requested from the agency, documents of the type that had been rou- 
tinely furnished the subcommittee on a regular basis involving many 
other countries (including Cambodia) for many years. 

Perhaps one logical explanation of this "adversary" position is 
that less executive-legislative branch cooperation can usually be ex- 
pected when the Federal Government is operating during a period of 
divided responsibility — one branch controlled by one political party 
and one by the opposite party — as has been the situation during the 
past 7 years. Many of these same observations can be made with re- 
spect to the adversary relationships between the subcommittee and 
executive agencies during the oversight and legislative activities 
involving the Freedom of Information Act. 

One effective technique utilized by the subcommittee during its 
investigation of AID program activity in Laos involved the use of a 
member of the staff as the lead-off witness to put into precise con- 
text the scope and detail of staff investive efforts. By narrative 
summary, he described in testimony the ways in which the various pro- 
grams functioned, presented evidence of waste and maladministration 
he had uncovered, and set the stage for subsequent cross-examination 
of responsible AID officials at another subcommittee hearing. 

Congressional oversight might well be divided into three main 
categories — 

1. "Routine" oversight , for example, would be the day-to-day 
monitoring of agency regulations as published in the Federal Register 



203 



to implement the 1974 FOIA amendments — contacting responsible agency 
personnel to call attention to technical errors, misinterpretations of 
congressional intent, or to request clarifying amendments to the regu- 
lations. 

Another example of a "routine" type of oversight would be informal 
briefing sessions of subcommittee members and/or staff with officials 
of an agency to become more familiar with a specific program, to be- 
come better acquainted with agency personnel, to clarify a policy or 
operational change of the agency, or as a preliminary step prior to 
the scheduling of oversight hearings with the particular agency. Un- 
less there was a major controversy involved, the subsequent hearing 
with agency officials would also be considered as "routine" oversight. 

2. "Special" oversight differs from "routine" in that it usually 
represents a major effort of the subcommittee in an important area of 
its jurisdiction that has a high priority. Special oversight work 
also indicates the need for a long-term series of investigations, hear- 
ings, sometimes requiring the use of expert consultants, General Ac- 
counting Office investigative work, staff liaison with the staffs of 
other House and Senate committees having collateral jurisdiction, field 
investigative trips, and other types of in-depth studies. It also 
means that the results of such "special" oversight will not only be 
valuable as part of a major investigative report of the committee, 
but are also likely to include longer-range legislative recommendations 
that relate to amendments to statutes falling within the committee's 
jurisdiction. Subsequent legislative actions to implement these recom- 
mendations would also be a part of the overall plan. 



204 



The case study of the 19 74 Freedom of Information Act amendments 
covering more than 3 years of oversight and legislative activity by 
the subcommittee would be a prime example of such "special" oversight. 

3. "Guerilla" oversight , the final category to be discussed 
here, is — as the title implies — a type of "spot checking" of mis- 
cellaneous activities by agencies coming within the scope of the sub- 
committee's jurisdiction. It differs from the type of "routine" con- 
tacts with agency officials mentioned above in that it will more likely 
represent an adversary or conflict situation. Often, the impromptu 
visit of a staff investigator to the office of an agency official, or 
a telephone call to discuss some agency program, decision, or other 
action of interest to the subcommittee can have a very salutary ef- 
fect. 

If additional impact is considered necessary, the subcommittee 
chairman might call a "1-day" hearing to follow-up the preliminary 
contact and to dramatize the agency shortcoming in that particular 
case. The resulting publicity may prompt quick remedial action by 
the agency. A formal committee report may then be issued to focus 
added attention of the Congress and the public on the agency's in- 
ept behavior. 



205 



DISCUSSION 

Several important points were raised in the discussion following Mr. 
Phillips' presentation. In summary these are: 

(1) Good oversight, and the information gathering and analytical 
skills which underlay good oversight, require very competent professional 
staff; 

(2) Good oversight generally requires the adoption of an adversary 
stance between the executive and legislative branches regardless of the 
bipartisanship of the issue or the fact that the President and the 
majority in Congress may be of the same party; 

(3) An analytical research organization which ascribes to the 
ethics of objectivity and non-partisanship generally cannot perform all 
the tasks required for good oversight, since oversight has to take place 
in the political arena. The appropriate role for CRS is as an information 
resource not an evaluator. Generally the smaller the congressional 
committee oversight staff, the larger the CRS assistance role in oversight. 

(4) The credibility of an investigative staff person is usually 
not diminshed if he or she delivers testimony before the congressional 
committee. However, investigative staffers should be able to weigh 
the information gathered according to both majority and minority points 
of view if these views differ. 

(5) Adequate routine oversight can be obtained frequently through 
normal responses to constituent inquiries. 

(6) Oversight would be improved if committee staff gave more atten- 
tion to the Federal regulations that agencies issue. It is important 

to assess whether the regulations correspond to the intent of the Congress 
in passing the mandating legislation for a program. Usually not enough 

attention is paid to regulations printed in the Federal Register . 



206 



"CONGRESSIONAL COMMITTEE CASE STUDIES: STUDIES IN PUBLIC WELFARE" 
ALAIR A. TOWNSEND, FORMER RESEARCH DIRECTOR OF THE SUBCOMMITTE 
ON FISCAL POLICY, JOINT ECONOMIC COMMITTEE. 



Thank you for the opportunity to discuss with you how and why the 
Subcommittee on Fiscal Policy conducted a comprehensive public welfare st' i; 
that resulted in the 24-volume series entitled Studies in Public Welfare . 
Origin of the Study 

The focus and nature of our study was shaped by a legislative debate, 
out 6f which it grew directly. The debate was over the Family Assistance 
Plan, the Presidential proposal first advanced in 1969 to establish a Federal 
minimum income floor for all families with children. As you know, former 
Representative Martha Griffiths, who was chairman of the subcommittee, also 
was a senior member of the House Ways and Means Committee, which handles a 
substantial portion of public welfare legislation. 53/ The deliberations o~ 
the Ways and Means Committee in 1970 and 1971 on the Family Assistance P__.., 
known was FAP, convinced Mrs. Griffiths and others of the necessity to a. ... „yi;e 
FAP and related plans in the context of all the existing income-Lasted c.i...) 
assistance programs, the in-kind programs offering food, health, day care 
and housing benefits, and the social insurance programs- — such ;:3 social 



* Ms. Townsend was Research Director of the Subconmittee en Fiscal Policy 
during its study of public welfare programs. She is not.- Senior Analyst - 
Human Resources for the House Budget Committee. Views express./! are the 
author's only, and do not necessarily reflect those of the Housj Budget '. . 
mittee, its members or staff. 

53/For Mrs. Griffiths' own statement of her concerns in conducting the st, ., 
see attachment 1, her "Foreword" to income Security fo r A-i cric.i ns. Her 
statement of appreciation also shows the range of assistance received cxi\. 
resources utilized in the course of tlie study. 



207 



security and unemployment insurance — rather than simply in the context 
of the existing public assistance programs. In the FAP debate attention 
was directed to the following problem areas. 

1. Adequacy of Benefits. First, in order to assess the adequacy of FAP 
benefit levels, policy-makers had to know what other benefits and private 
income were available to potential FAP recipients and how they were dis- 
tributed. Lacking this information, one could not know the combined benefit 
and income levels that were implicit in the legislation. For example, the 
original FAP basic cash benefit level, $1,600 annually per family of four 

in 1969, appeared to be cruelly low to some, but for many recipients this 
was to be supplemented by State welfare payments and by the benefits thac 
usually are attached to enrollment in the program of Aid to Families with 
Dependent Children (AFDC). These fringe benefits include free medical care, 
free school lunches for the children, eligibility for food stamps or surplus 
food commodities, and, often, free day care. Depending on the State in 
which it lives, a family of four with no other income now can receive a 
package of benefits that would require the equivalent of more than $7,500 in 
gross earnings (that is, earnings before taxes and work expenses are paid) 
to purchase. Of course, AFDC families in many areas receive nowhere near 
this amount, but during the FAP debate no one was very certain who received 
what because benefits in the form of food, health, and housing rarely are 
included in official census-style surveys, and measurements of the extent 
of poverty do not count the value of such benefits in summing the incomes 
of the poor. 

2. Work Incentives. Another reason for evaluating refor.n plans such as 

FAP in a broad programmatic context is that the level of combined cash, fooo, 



208 



housing, health, and other benefits is only one factor in determining the 
effect of such plans on work incentive. Equally important is the way in 
which benefits are tailored to income. 

Today, many programs have sliding scales of benefits that fall as 
income rises. Some of these benefit-reduction or benefit-loss rates are 
high and some are low. Unemployment insurance typically has a 100 percent 
benefit-loss rate — that is, a dollar earned is a dollar of unemployment 
compensation lost. AFDC benefits generally are reduced by 67 cents for each 
dollar of net earnings. As income rises by $1.00, you must pay about 30 
cents more for the same amount of food stamps and about 25 cents more in 
public housing rent. If income rises above a certain level, you may lose 
eligibility for medicaid completely. The cost of day care may increase if 
the center bases its fee schedule on family income. The basic opportunity 
grant program also provides scholarships based on family income, and the 
higher the family's income, the lower the aid. 

The number of programs based on need seems to be increasing, and, thus, 

the number of low- or even middle- income families eligible for multi-benefits. 

Since each program takes its share of increased income, the cumulative 

effect sometimes is that recipients lose more than a dollar in benefits 

when they earn an extra dollar. 

If simply superimposed on existing programs, new proposals worsen this 

problem. The Family Assistance Plan suffered fatal attack after it became 
clear that it both raised the guarantee level for penniless persons receiving 
various combinations of benefits, and more importantly, raised to extraordinarily 
high levels the rate at which combined benefits would have to be surrendered 
if earnings rose. The work incentives so carefully built into t"AP would neve 
been undermined by the program's interaction with others. 
3. Equity of Benefits. The equity of benefits became another issue in the 



209 



FAP debate. The distribution of existing benefits, especially those tied 
to eligibility for AFDC, was unfair. Most of the cash aid was unavailable 
to "employables," no matter how poor they were. Fully employed fathers, 
for instance, were ineligible for AFDC, even though such families were a 
significant prart of the poverty population. The Family Assistance Plan, 
by opening up Federal cash aid to intact families, promised to reduce the 
discrimination against male-headed families. But, critics countered, as long 
as FAP benefits were lower than the AFDC benefits provided by some States, 
and as long as eligibility for other benefits such as medicaid was tied to 
AFDC, inequities could be reduced but not eliminated. 

That criticism was valid but it was not generally appreciated how 
much male-headed families already were receiving in benefits to supplement 
low income. In fact, some States and localities were supplementing the 
earnings of working fathers under their own general assistance or home 
relief programs; such working men were eligible for the food stamp program, 
which acquired national eligibility rules during the FAP debate and expanded 
to cover most of the Nation. Intact families were eligible for public 
housing and other subsidized housing programs on a space available basis 
like everyone else; and in some States they — or their children — were 
eligible for free medical care under medicaid. If they, were employable but 
unemployed — whether fully or partially — many were eligible for unemployment 
insurance or for the unemployed father portion of AFDC, available in about 
half the States. However, the extent to which such programs helped to reduce 
the inequities created by AFDC or potentially residual under FAP was 
unknown because little data existed on who was benefiting from these programs. 
Thus, to judge the effect of FAP on equity of benefits as well as their 
adequacy, new data were needed. 



210 



A. Administration and Program Planning. The problem of administering 
this complex set of programs came into focus in the FAP debate. Rapid 
expansion of such programs as AFDC and food stamps was outpacing the ability 
of administrators to handle them within existing organizational structures. 
Administrative confusion was created by the complex, ever-changing rules 
and eligibility requirements. Result: rather substantial numbers of in- 
eligible persons received benefits while some eligible persons were rejected. 
While each program had its own problems, administration of multiple programs 
at the local level generally was uncoordinated. A single family could be 
affected by several programs, none of which shared information or adminis- 
trative burdens. The result was inefficiency and confusion. Rather than 
cutting and paring, it appeared to many that FAP was only adding another 
layer. They wondered: "Can't we develop a plan that can consolidate and 
substitute for some of those that currently exist?" 

5. Need for Overview. Despite these complex interactions among our growing 
body of programs designed to affect standards of living, neither the structure 
of Congress nor of the Executive agencies facilitates their analysis. Con- 
siderable analytical work is done by congressional committees, the Executive 
agencies, and nongovernmental researchers before major legislation is changed. 
But in general such analysis focuses exclusively on one program at a time — 
the one under consideration. Rarely is the set of programs reviewed in its 
entirety. The result is that intended changes in one program often are 
countered by features of an overlapping program. For instance, the work- 
refusal penalty of the AFDC program — a reduction in cash benefits — is largely 
cancelled, in some States, by higher food stamp benefits. 

The Congress and the Executive agencies are organized in programmatic 
terms and have jurisdictional boundaries. For example, if we look only at 



211 



the 21 major income maintenance programs with some Federal funding, we find 
they fall under the jurisdiction of 10 committees of the House and 9 of the 
Senate, and they are administered by 11 Federal agencies. No single com- 
mittee has the duty to view them all, even though the programs reach many 
of the same persons. Each passes legislation affecting directly only a few 
programs with different eligibility requirements, different benefit levels, 
different treatment of income. 

Proliferation and fragmentation of programs have created serious 
technical and policy problems, but they went unrecognized until the FAP 
struggle. Once the FAP debate raised such issues in the Congress, it ap- 
peared that one committee was well suited to review the entire complex of 
programs cutting across many jurisdictions. This was the Joint Economic 
Committee, which is non-legislative and has a broad study mandate. Mrs. 
Griffiths and the Subcommittee on Fiscal Policy received a separate appro- 
priation to conduct this study, and report findings to the full Congress. 
The major rationale for this study was to determine ways to integrate and 
rationalize this growing array of income-related benefit programs. 

To summarize the background of the subcommittee's study, our task 
was to go beyond the one-at-a-time program review generally performed. We 
were to review the entire set of income support programs and report to the 
Congress on the effects of these programs with regard to: 

1. The distribution of benefits, especially in various combinations; 

2. Questions of adequacy; 

3. The impact on incentives with respect to family structure and 
work effort; and 

A. Administrative efficiency. 



212 



The Product 

The study lasted for three and one-half years. Its product was twenty- 
four volumes of staff-level studies, four volumes of hearings, a summary 
report with recommendations endorsed by the majority of subcommittee members, 
a reform proposal in bill form, and numerous articles published under other 
auspices. Several of the most innovative projects had been conceived in 
advance by Chairman Griffiths. She knew well what the information gaps 
were for policy-makers, and had outlined in some detail particular studies 
necessary to fill them. Throughout the study, Mrs. Griffiths' guidance and 
practical knowledge of programs, issues, and how to present information 
were extremely important to staff. Her accessibility and direct participa- 
tion in all phases were key elements in maintaining policy relevance. 

While the chairman had presented a working staff study plan to sub- 
committee members in the early stage, many projects simply evolved during 
the study period. There was sufficient flexibility and time to permit 
follow- through on good ideas. 
Resources 

Studies of the scope intended here, on complex programs, require 
commitment of resources, time, and patience. It takes time to assemble a 
staff and to collect and compile information to do a thorough jcb. In the 
case of the subcommittee, the initial funding was for two years, extended to 
three, then to three and one-half years. 

The staff was a carefully selected mixture of a variety of skills ar.d 
disciplines. It was essential, for example, that there be several persons 
with strong institutional knowledge, who knew in detail the laws and 
regulations under which the major programs were administered, and who also 



213 



knew how the programs actually operated in the field, since frequently there 
is a wide gulf between formal rules and their application. Similarly, we 
needed persons with hard research and quantitative skills, who could help 
design surveys and analyze data. The staff approached most projects as a 
team, and the various skills and perspectives of individual staff members 
strengthened each project. 

A strong staff was essential in directing and reviewing work done by 
outside researchers free or under contract, and in monitoring projects 
undertaken for us by Executive agencies, CRS, and GAO. Proper control of 
a project, in my view, requires a strong committee staff — not to control 
findings , but to retain the desired focus . These "outside" (non-committee 
staff) resources are not limitless, and should not be squandered by lack 
of direction. 

Other than staff, our major resources were the GAO, CRS, outside 
researchers, State and local program agencies, the House Information Systems 
staff, and Executive agencies. In any given project that involved non-staff 
resources, the approach was to have a staff specification of the project and 
the help required, followed by regular monitoring. 

One of our major projects, for example, involved our staff, the GAO, 
and the House Information Systems staff (HIS) . The goal was to learn more 
about the distribution of benefits, and combined benefits in particular. 
To do this, the staff and GAO chose six low-income study sites. Utilizing 
city block maps and other available resources, the GAO selected, in each 
site, a random sample of about 300 households. Combining use of telephone 
and post office directories with a good deal of imagination, -GAO staff 
identified the members of the sample households. Then, largely without con- 
tacting the households, GAO searched the records of over 100 Federal, State, 



214 



and local programs to determine whether any household members were partici- 
pating. They combined these facts by household to produce a program 
participation profile. (The results were dramatic. Findings showed one 
family benefiting from 11 programs; many households were participating in 
three to five programs.) GAO staff coded and keypunched the data, House 
Information Systems processed it, and our staff wrote the report. 54/ All 
along the way there was interchange between GAO and subcommittee staff, and 
the division of labor wo-rked well. 

In another case, we wanted to learn more about the actual rules applied 
by States and localities in programs such as Aid .to Families with Dependent 
Children, supplementation for the aged, blind, and disabled, public housing, 
general assistance, and medicaid. These are programs in which States and 
localities have considerable discretion. Additionally, we wanted to see 
what benefit packages might be available to various family types at various 
income levels in areas across the Nation. Such information did not exist. 
So we devised a questionnaire to be filled out by State agencies (or their 
local designees) on programs operating in 100 counties selected on a 
stratified random sample basis to be representative of the Nation. All 
relevant assumptions were given, such as age and sex of family members, their 
work expenses, their income, and so forth. Ten family types and about 6 income 
levels were specified. Before sending out the questionnaire, our staff com- 
puted and filled in Federal, State, and local income taxes and social security 
taxes if applicable; Federal benefits such as social security and veterans' 



54 / See Paper No. 6 as listed on attachment 2. 



215 



pensions if applicable; unemployment insurance benefits (with the help of 
the Department of Labor); public housing benefits (with the help of the 
Department of Housing and Urban Development); and so on. Completed 
questionnaires were reviewed by staff, processed by House Information 
Systems, and reports written by our staff .55/ These two reports — one covering 
programs for the aged and the second for the non-aged — were widely cir- 
culated within Congress. Detailed information on programs in their 
counties or States proved to be of great interest to Members. 

Our concern with the problems generated by insufficient program 
coordination naturally led to thinking about how to improve matters. In 
conjunction with the Institute for 'Research on Poverty of the University 
of Wisconsin, the subcommittee sponsored a week long conference on program 
coordination attended by subcommittee staff, institute researchers, other 
experts in the field, and, for several days, Mrs. Griffiths. Mot;t partici- 
pants were asked to write papers about specific existing programs, later 
edited and published by the subcommittee. 56/ To assure maximum usefulness 
to the subcommittee staff in developing reform options for the chairman, 
the papers were to cover at least the following topics: a description of 
the program as it currently operated; the role it played in the income 
maintenance system; how well or how poorly it meshed with other programs; 
and optimal restructuring required under each of three specific major 
welfare options — a deraogrant, an employment-based program, and a negative 
income tax-type plan such as the Family Assistance Plan. 



55/ See Paper No. 10 and Paper No. 15. 

56/ See Paper No. 4, Paper No. 5 (Part 2), Paper No. 7, Paper No. 9, and 
Paper No. 19. 



71-867 O - 76 



216 



Another use of outside resources involved research and evaluation 
studies of manpower training programs. Many such studies of program effectiveness 
had been conducted, largely funded by the. Departments of Labor and HEW. But 
a variety of methodologies was involved, findings sometimes were contradictory, 
and no general summary was available. One subcommittee staff member under- 
took the task of critically reviewing each study to determine the overall 

impact of these programs on the poor. 57/ 

A final type of project involving non-staff resources was compiling 

a handbook of public welfare programs. Very early in the study, we 

recognized the need to have in one volume, in easily accessible form, detailed 

information on the rules under which current programs operate, especially those 

features that were important in considering questions of program coordination. 

So we developed a standard format to be used for each of the major cash and 

in-kind programs that would specify such details as conditions of eligibility; 

the eligible units (e.g., primary beneficiary only, household, family, etc.); 

definition of personal income relevant for program purposes; hov* that income 

is counted in determining eligibility and benefit amounts; State 

variations in eligibility and benefit provisions, and other such items. Our 

primary sources, of course, were the law and regulations. The result was a 

volume heavily used by our own staff and, we are told, by many others. 58/ 

Programs change constantly, however, and portions of the Handbook 

were out of date as soon as it was published. At the end of the study we wantta 

to leave an updated version, but our time and resources were limited. 

Staff revised our fairly detailed format. Staff, CRS, and 

the Executive agencies updated the H andbook , using this outline as a mode] .59/ 

57/See Paper No. 3. 
58/See Paper No. 2. 
59/Sec Paper No. 20. 



217 



These examples of our collaborative efforts were intended to make 
two points. First, there is no substitute for a highly professional staff that 
knows the substance and programs under review. Second, outside resources will 
be best utilized when the staff has clearly specified what it wants done 
and follows the project closely. 

As I have mentioned, we made extensive use of the published (and in 
some cases, unpublished) data from Executive agencies and the considerable 
program expertise of Executive staff. I think it fair to say, however, that 
the original data we collected on programs made it a more than even 
trade. In several cases, agencies collected new data on their programs 
and participants directly at Mrs. Griffiths' request. xhe Department of 
Agriculture, for example, undertook the first in-depth survey of the 
income and other benefits received by households participating in the food stamp 
and surplus commodity programs. 60/ 

Presentation of Findings 

Doing a good study is one thing; presenting the information in useful 
formats is another. The. work has to be sufficiently technical and 
scholarly to be credible among researchers, academicians; and experts in the 
field. But, as congressional staff, these groups were not our principal 
audience. To reach Members, their staffs, and committees, it was necessary 
to summarize results in non-technical terms. Every subject matter has its 
own technical jargon, which can be a barrier to capable, interested persons 
who should not have to take the time to learn it. Since it is too much to 
expect each long, detailed report to be read in its entirety by each 



60/ See Paper No. 17. 






218 



interested. person (although I feel strongly that the details have to be there), 
we made use of chart books, detailed press releases generally sent out with 
publications to non-press as well as press, and summary volumes. We had 
published two extensive volumes on the subject of work incentives, for 
example.—'' But the subject was too important not to attempt a short, 
summary volume with many charts and graphs. 62/ 

SUHHAgY 

To summarize, in the case of the subcommittee, there were three elements 
for a successful study: a chairman interested in and knowledgeable about the 
subject matter, sufficient time to do a thorough job, and a strong (though 
not large) staff. 

It is too early and quite difficult to determine the impact the 
study has had or will have on legislation. Two facts are relevant: first, 
many Members, committees, and staff of both houses, are familiar with the 
study; second, the study volumes were widely distributed and even used as 
textbooks in a range of disciplines. In terms of helping to fill information 
gaps, those are large steps. Moreover, there is much wider recognition 
that individual cash, food, housing, health, day care, and scholarship 
programs are simply pieces of a larger income maintenance' system. There 
is a new appreciation of the difficulty of reconciling work incentives with 
adequacy of benefits in income support operations. If it be true that 
recognizing and measuring a problem is the first step on the road to solution, - 
the subcommittee study has made a substantial contribution to a better way 
of helping the poor. 



61/See Paper No. A and Paper No. 13. 
62 /See Paper No. 14. 



219 



ATTACHMENT 1 



FOREWORD 



Before I close (lie books on this survey of income maintenance pro- 
grains and how they work, I would like to take, this last opportunity 
to thank those who have so kindly helped me. cither to do the work or 
to secure the appropriations which made the study possible. The Con- 
gressmen are: Wright Patman of Texas, Wilbur 1). Mills of Arkansas, 
ex-Congressman Johnny Byrnes of "Wisconsin. George H. Mahon of 
Texas, the late Frank Bow of Ohio, Gerald R. Ford of Michigan; tho 
Senators are: William Proxmire of Wisconsin, Jacob K. Javits of 
New York, Abraham A. Ribicolf of Connecticut, and Finest F. Rol- 
lings of South Carolina. I want expressly to thank the subcommit- 
tee memhers for (heir encouragement in (he effort and the subcommit- 
tee stalf, both current and former members, for the excellence of their 
work, their patience and their good humor. It was a pleasure to work 
with them. 

In addition, this study would never have been possible without the 
devoted and" untiring efforts of my favorite, agency, the General Ac- 
counting Oilicc. nor could this study have been completed without tho 
tremendous help of welfare directors and workers throughout the 
country, particularly those who aided us in the survey of benefits in 
100 count ies. and the work of those researchers who submitted papers. 
I want also to acknowledge with gratitude the work of the following 
persons and organizations: Robert llaveman, director of the Institute 
for Research on Poverty, University of Wisconsin: Irene Lurie. co- 
ordinator of a Conference on Coordinating Income Maintenance Pro- 
grams, sponsored jointly by the subcommittee and I he Institute for 
Research on Poverty; the IIouso Information Systems staff; the Con- 
gressional Research Service, particularly Robert Boslick who pre- 
pared charts for the subcommittee, and the. Education and Public 
Welfare. Division: Donald Craver of the Government Printing Oflicc, 
who expertly handled the printing of subcommittee volumes; stall's-Of 
tho Departments of Health, Education, and Welfare: Agriculture: 
Housing and Urban Development; and Labor; Robert Lampman and 
Harold Walls of (he University of Wisconsin: Robert Harris. Deo 
Bawden. and Gary Hendricks of the Urban Institute; Nelson Mc- 
Clmig. Treasury Department: David Kershaw. Mathematical Theo- 
dore Marmor. University of Chicago; John Stark, executive director 
of tho Joint Economic Committee: and James Knowles. former re- 
search director of tho Joint Economic Committee. My deep apprecia- 
tion goes to all of those named and many moro who aided me in com- 
pleting this historic work. 

Perhaps the first question that should be answered is why (his is (ho 
first congressional Study of welfare and related programs. That ques- 
tion I cannot answer. 1 oitlv know why I believed that one was neces- 
sary and why 1 undertook it. ' ■ i 

(V) 



220 



VI 



When I became a member of the Ways and Means Committee in 

lP(i2. I was first struck by I lie incredible inequities of social security, 
which returned on the same tax payment a greater sum to a man-sup- 
ported family than to a woman-supported family and more, to many 
families where, the woman remained at home than to families whero 
both husband ami wife worked, although the latter families might in 
.fact have paid more tax than one-worker families. 

Years ago I held hearings on private pensions, and retirees drilled 
into my head the fact that through quirks in the. labor contract or 
oAif right dishonesty in the pension setup, they had been deprived of 
their pensions. Of course, the Nation had been deprived of the tax 
money that should have been paid on those pensions, and now the, 
. Nation maintained these people on welfare. 

I watched far whatever else I could find on the law's strange 
workings. 

Some, years airo, I received a letter from a woman in my district 
who earned Sfi.^OO gross per year. She outlined her taxes, her take- 
home pay and her problems. The. next day T received a call from a 
woman living in a far better area of my district whose home had been 
purchased under section 235 and who was drawing, she said, w £750 per 
month, untaxed, in AFDO money, including $-2<>0 per month for a 
housekeeper. My "tired lady", earning $5.-".00. was paying taxes to 
he!]-) support a woman at a $7.">0 rate per month. T seethed. There are 
no jobs of which I am aware that increase your pay because you have 
children. 

Shortly after that day T began to investigate the possibility of 
making a complete survey of all income maintenance programs. 

"When the staff had been assembled I was invited to make a speech, 
on welfare to a church group in Detroit. T asked my staff to tell me 
what a woman with three children could get, in welfare, food stamps, 
medicaid, and other supplements in my city. Detroit, and in New 
York, Chicago. Atlanta, and Los Angeles; then to find the median 
earnings of a woman in those cities. We found, of course, that the 
woman on welfare did better in every city than the woman Avho 
worked at the median wage. 

I will never forget some of these hostile questions in that church: 
''But the "woman must pay some money for tbe food stamps." Answer: 
"We have considered that. She is still getting a substantial amount of 
free food." Question: "But she waits all day for a doctor." Answer: 
"Who doesn't? But the. woman who works may give np a day's pay, 
wait all day. pay :?lo to the doctor and $50 for prescriptions." Ques- 
tion: "But she has to take three buses to iret to the place to <rct the 
food stamps, welfare, etc." Answer: "The woman who works may 
take three buses twice a day, 5 days a week to get back and forth to 
work." 

The. theory of comparing what is given in welfare with what is 
needed is foolish. "What is needed" is a phony standard set np by a 
paternalistic middle, class. The real standard is what, similar people 
earn, and how they are treated. Few have ever asked what those, who 
work need. Those who have bled for people who have nothing have 
not demanded that people- who work be treated at least as well as 
people on welfare. 



221 



VII 



In the coming effort to correct the "welfare" injustices there will 
be those who say that however much is offered is too little, and those 
who say that however little is offered is too nnich. T urge boih groups 
to consider the real world and to aslc themselves: "Can we all'ord this 
for all in like circumstances, or ain I, today, selecting out one small 
group?" 

Personally, I am for the law applying equally to all, and this is the 
purpose of the entire investigation of income maintenance programs. 

This work began because of the inequity in the law. It is dedicated 
to equality in justice. 

Martha "W. Grietitiis, 
Chabinan, Subcommittee on Fiscal Policy. 



222 



ATTACHMENT 2 



PUBLICATIONS OF TUB SUBCOMMITTEE ON 
FISCAL POLICY 



Papers in I be Series :8tmHes fothe Public Welfare 

PaiKT N... I. "Public Income Tr.-in-.fer Pre-grants; The incieleiiee- of Millli|il.! 
He-iicnls :in«l tin- l--ue- llai-.-.l liv Tin ir lle.-.-ipt." A|ir. III. l'.IT'J. 

l*:ip.-r X... J. "Il:iiull I; r.r Public Income- Trni.-f.-r Program-," Oct,. III. l'.IT'J. 

Paper X... 3. "Tli.- i:ir.'.Miv„ f Manp.oe.-r Training Program-: A P.e-vicw „f 

Research on ili<- ln.|.:>n on tin- l'.».r." Nov. JO, l'.IT'J. 
Paper Xo. 1. "Income- Trai.-fe-r Pro-rains: How Tlu-v Tax the; Poor," Deo. 22, 

1*172. 
Paper No. ... I-<u.- in Welfare- Administration: 

(Part 1) "Welfare— An AdminMraiivc Nightmare," Dec. :il, 1072. 

("Part 2) "Inti-raoviTiiiiii-inal lSi-lailoii.hips," Mar. 12, PIT:!. 

(Pnrl_3) "Implication- of the Income Maintenance l-Ape-rliiienls," Mar. 12, 

Paper Xo. C "II. .n- Public Welfare- Benefit* Are Distributed in I.ow-Incomu 

Areas," Mar. 2(i, HIT:!. 
"Additional Material for Paper Xo. (i: How Public Welfare Benefit* Are Di* 

tribuled in ijrtoliicomc Ana-." Aug. (i, HIT3. 
Paper Xo. T. "Issues in the Coordination of Public Welfare Programs," July 2, 

iot:;. 

Paper Xo. S. "Ineonie-TeMed Social licni-ltls in New York: Adeepiiicv, Incentives, 

and Equity." July !<. I!l7:i. 
Paper Xo. !>. "Concept- in Welfare- Program D.-ign," Aim. 20, 10T3. 
Pape-r Xo. 10. "The- Xi-iv Supple-menial Se-cnrilv Income- Program— Impact on 

Ciirre-nt lU-n.-lii- anil I'lin-oh ,-,l l-sii.-s." Oct.' 7, KIT:',. 
Paper Xo. 11. "The Labor Market Impacts of the Private P.etircinent System," 

Oct. SO, 107:!. 
Paper X... 12. The Familv, Poverty, and Welfare Program*: 

(Pari I) "Factors Inilueiiriug Family In-ial.ilil v." X.iv. 4. l!IT:t. 
(Part II) "Hoii-chold Patterns and ( '...verniu.nl Polici.-." Dec. 3, 10T3. 
Paper Xo. 13. "liow Income Siipplciiuuis Can Allecl Work ISehavie.r," Feb. IS, 
1074. 

'Public Welfare- and Work Incentives: Theory and Practice-," 



Paper Xo. 14. "Publi 
Apr. 13, 1074. 



'Welfare- in tin- TO's: A National Studv of Hem-fits Available- in 
100 Local Areas," Julv 22, 107-t. 
Pape-r Ne.. Hi. " A Model liieome- Supplement Kill." I )e-c. 20. 107-t. 
Paper No. IT. "National Survey of Food Stamp and Food •Distribution Program 
Re-cipie-nts: A Summary of Findings on Income Se.urce.s and Amounts and 
( .Incidence- ..( Multiple Benefit*," Dec" 31, 1074. 

Paper No. 18. "Issues in Financing Retirement Income," December 27, 1974. 

Paper No. 19. "Public Employment and Wage Subsidies," December 30, 1974. 

Paper No. 20. "Handbook of Public Income Transfer Programs: 1975," December 31, 1974 

Report, "Ine le- Security for Americans: Re coiniiicndnlinn- of the- Public 

Welfare- Study," D.-c. 5, 1974. 

1 I.-niii.-s in I he Series : I'.ohh,,,, n, .Llmieixtmtia* of Public- VT< Ifvrt 
Profftuhta 

Pari I. Mar. 20, 1072, Wasllillgli.ll, D.C: and Apr. 11, 12, and 13, 1072, New 

York, NY. 
Part 2. Mav 3, I, ami ... l'.IT'J. Detroit, Miclu 
Part 3. June li, T, and S, l'.iTJ. Atlanta, lia. 

Other HnmngM ' 1 

Open-laid. -el Federal Matching of State Social Service i:\p.-niliiuiv Anihori/.-d 
Und.-r I he Public A-i-iaoe-i- Till.- of I lie Social Serurilv Acl, S. pi. 12, 13, and 
II, H.I72, Washington, U.C. 



223 



DISCUSSION 

During the discussion it was noted that the Subcommittee was sub- 
ject to some interest group pressure while doing the study. The type and 
intensity of outside attention varied depending upon the nature of the 
issue or program being studied. Some programs were very popular and 
study of them generated resentment, for instance on the work training 
programs. Not all outside group pressure was necessarily dysfunctional. 
Usually the committee did not encounter external criticism. This may 
have been due to the fact that the committee did not have legislative 
authority. Another reason may have been the fact that rigorous research 
and analytical methodologies were used, averting possible criticisms about 
inadequate quantitative research designs and analysis. 

Several points were raised about the administrative costs and bene- 
fits of public welfare programs. The subcommittee discovered that ad- 
ministrative costs vary from two to twenty percent of the program costs. 
Generally only a small fraction of actual program costs are used on 
data gathering, research, and evaluation. However^ much of this data 
is useless for good evaluation. 

Some discussion focused on the organization of program evaluation 
within agencies. The subcommittee concluded that evaluations should be 
sponsored by an office attached to the Office of the Secretary of an 
agency since this insures better access to decisionmaking. 

During the discussion, Ms. Townsend elaborated upon a point she 
made in her formal presentation. Committee staff found it necessary to 
monitor very closely the work of their outside consultants, especially 
social science researchers. Even though researchers are on contract to 
answer specific policy questions, they sometimes orient their work more 

to meet other goals, including basic research and teaching. 



224 



'POLICY FORMATION THROUGH PROGRAM EVALUATION AND SYSTEMS ANALYSIS: 

A CONGRESSIONAL VIEW," JAMES W. GILTMIER, SENATE COMMITTEE ON 

AGRICULTURE AND FORESTRY 



Ten constituents write, "We want more wilderness," and a congressio- 
nal eyebrow lifts. One hundred more write and a staff project is initi- 
ated. The legislative process — the business of policy formation — is 
in action. It is as difficult to stop as it is to start. 

Where will it lead? Who knows? 

It's simple enough. Find out what all sides think; run it by the 
Administration to get feelers on a veto; make a separate check with the 
Office of Management and Budget to determine what that secret government 
might think; draft a final bill; run it past a subcommittee, than a full 
committee for refinement; pass it in the Senate; adopt the conference 
report; and finally send it to the White House for the President's sig- 
nature. 

Throughout this process, lawyer-lobbyists on both sides of a ques- 
tion, as well as congressional and executive department lawyers are mov- 
ing the commas and parentheses around in mystical ways. Often the 
process becomes so cluttered and complicated that it is difficult to re- 
member that "all you wanted to do was drain the swamp," or create a 
wilderness. 

The product of all this is that which is politically possible. It 
is not always good. Often that product is an overreaction to a situ- 
ation, but this is often resolved because not enough funds are appropri- 
ated to allow the overreaction to cause too much damage. 

It's a sloppy, unscientific system which in no way lends itself to 
a systematic approach. But it's democratic, and that's what really counts. 



225 



The Founding Fathers didn't promise us a rose garden, but they did offer 
the people a voice in where the weeds would be permitted to grow. 

Isn't it odd that the compromise and bargaining required to get a 
measure into law is the very thing that many Americans detest most about 
our political system. Yet the democracy could not survive without the 
push and shove of interests against one another until consensus is found. 
Sometimes these diverse interests are regional, sometimes economic and 
sometimes social. But it's important in adopting legislation, and there- 
by setting national policies, that no one's ox get gored too much. It's 
bad enough in the minds of many Americans that there must be a Govern- 
ment at all. Therefore, Government excesses could tend to lead to a 
revolution. 

Nevertheless, the United States is beginning to get a little age on 
it. Congress is a bit more mature in the process, and there is a large 
body of public law that has been passed. Many of the members of our 
policy-making arm of Government are wondering if law is not passed by 
listening to cases being overstated on either side of issues, without 
any concern for whether previous laws handle the situation very nicely, 
thank you; whether the executive agencies have the capacity to carry new 
laws out, given staff and funding levels; or whether the good intentions 
in law have anything to do with the delivery of services to the public 
at the local level. For that matter, there are questions being raised 
as to whether Congress is adequately staffed to know what happens after 
legislation is adopted, or whether it is equipped to determine in advance 
what the actual effects of new legislation might be. 



226 



A sensitivity to all of this led Congress to adopt the Legislative 
Reorganization Act of 1970 — which in itself was quite a compromise, 
and according to some elements of the press, not much of a reorganiza- 
tion at all. 

Section 204 of the Act directs the General Accounting Office, either 
on its own initiative or at the request of committees of the Congress, 
to make studies of the costs and benefits of existing programs. 

Clearly the intent of this is to see if we have any old laws hang- 
ing around that aren't much good, or are costing the taxpayers more than 
they should for the benefit received, and the process should also be 
used for a quick informational turnaround on whether the executive agen- 
cies were carrying out the intention of Congress. 

Now this is clearly heady stuff. Implied here are qualifiable and 
quantifiable measures of program effectiveness and maybe even a systems 
approach to policy formation. That could mean computers, information 
specialists and tons of paper spewing out the "who-shot-John" of govern- 
mental and maybe even congressional effectiveness. Obviously there are 
implied dangers too from "business as usual" bureaucrats who might not 
particularly care about what the wishes of Congress might be, new laws 
or not. 

(The GAO authority was significantly strengthened last year by en- 
actment of the Congressional Budget Impoundment Control Act.) 

It would only be fair to say that the GAO and congressional response 
to the Congressional Reorganization Act was timid. Neither side wanted 
to move too fast for fear that a gigantic structure be created in 



227 



Government that might amount in the end to a scandal-ridden boondoggle 
that delivered too much information to be usable by anyone, while cost- 
ing millions of dollars. 

Dr. Ida R. Hoos points out in Datamation: "Gargantuan systems that, 
with the ingestive propensities of snakes, have subverted and swallowed 
up the very organizations they were purported to serve are the outcome 
of a chain of apparently innocuous circumstances. Electronic data pro- 
cessing introduced in the mid-1950' s to hasten the paper flow, reduce 
clerical costs, and increase efficiency in information handling was the 
first step. The logic was clear, economic justification unassailable, 
and the role for an information technologist insured. 

"But two decades and two computer generations later the promises 
are still illusory. To be sure, records move faster, but their volume 
has increased exponentially. In a Parkinsonian fashion, they have ex- 
panded to fill all computer time. Clerical costs soar; the price of 
processing paper is rising astronomically but invisibly, as any accoun- 
tant charged with allocating company expenses can attest. "53/ 

Whoa! Wait a minute. Maybe buttonholing is a better way to make 
policy than by creating "management information systems." As Senator 
Henry Bellmon of Oklahoma is fond of saying, "If you like either sausage 
or law, don't ever watch either of them being made." But the Government 
already costs more than $300 billion a year, and no one knows where the 
money will come from for such fixed governmental costs as civil service 



53/ Hoos, Ida R. Can Systems Analysis Solve Social Problems? Datamation, 
June 1974:83. 



228 



and military pensions in ten or fifteen years. Maybe the Congress ought 
not enter into something that could be a very large ticket item without 
some assurances of extensive payoffs. 

On the other hand, Congress is timid about getting into something 
that was already installed by former Secretary of Defense Robert McNamara 
as a major artery in the operation of our military establishment. 

Even though McNamara' s systems proved on many occasions to be fatal- 
ly fraught with the garbage-in, gospel-out syndrome, Congressional de- 
mands to "prove" why the defense budget should be so high required 
systematic goal setting and evaluation. It's now a big business at the 
Pentagon. It is growing in the other executive agencies, even though 
one official in the office of Secretary of Defense concluded to me that 
while program evaluation was sophisticated and useful, it often tended 
to take the course of least resistance, supporting previously foregone 
conclusions. 

I don't know, but I suspect that the General Accounting Office got 
its early strength from the Al Capone case. You'll recall that even 
Elliot Ness couldn't get Al into Alcatraz for mutilating the bodies and 
lives of his enemies. It was the accountants over at the Internal Rev- 
enue Service who nailed him. This led, I believe, to the generally still- 
held conclusion that today's big-time crooks are too smart to be caught 
with guns. It takes accountants. Isn't that why the FBI's list of ten 
most wanted generally includes only psychopaths and poverty-stricken 
nobodies? 



229 



No one has ever been able to explain to me whether GAO is an arm of 
Congress, or whether it's some separate, super-snoop investigating agency, 
held in check only by its conservative and thoughtful Comptroller General, 
Mr. Staats. While it serves Congress faithfully with its carefully worded, 
eternally reviewed reports, more than 70 percent of its work is inspired 
from within the agency. Most often in the past, the GAO approach to an 
investigation was the auditor's approach. That is, are those bureaucrats 
out there doing their job in the most cost-effective manner, and is this 
program really necessary? 

That's fine, except Congress is often not interested in whether a 
program is cost-effective in the most literal sense, or whether a pro- 
gram is needed. Government, after all is to do those things for a na- 
tion's citizens which they want done but which they cannot do for them- 
selves. And when the GAO questions the efficacy of a program, the 
organization is questioning the wisdom of Congress in setting up some- 
thing in the first place. 

It's a sensitive business as GAO recognized in its first attempts 
to make the evaluation process more consistent with the Defense Depart- 
ment approach. Take for example this language from a "Discussion Paper 
on Principles and Standards for Analysis and Evaluation" drafted by GAO 
staff: 

Principles and standards for professional practice have 
existed for many years where it has been important to protect 
both public and client interests by the use of generally ac- 
cepted measures of competence. Examples are the practice of 
public accounting where the financial interests of investors 
are at stake; and the practice of engineering where the safe- 
ty of the public is at stake. More recently, there is a gen- 
eral recognition that major decisions regarding the uses and 



230 



amounts of funds allocated may have very important implica- 
tions for the general public as well as the direct recipi- 
ents of the programs. Since these decisions have been 
increasingly influenced by program evaluation and program 
analysis, it follows that there is a need for generally 
accepted principles and standards to govern this work. 

In the absence of such principles and standards, a 
number of problems are created. For example, the Urban 
Institute reported that: 'Two Job Corps evaluations, one 
using 'no shows' as a comparison group, and the other using 
later Job Corps enrolles as a comparison group, reached 
widely different conclusions on the value of the Job Corps; 
the first study estimated benefit-cost ratios of 4.5 to 5.0. 
This led one observer to note: 'This is a disturbing situ- 
ation where evaluation relating to the same reality may vary 
from submarginal to excellent, depending on the assumptions 
and attitudes of the analyst.' 

The discussion paper then goes on to suggest an outline of the sub- 
jects to be covered in an orderly set of principles and standards. The 
major outline headings are: Purpose of Analysis and Evaluation; Evalu- 
ation Methodology; Reporting Results; and Administration of Analysis . 
and Evaluation. The paper makes the point that "studies are difficult 
to use for congressional decisions unless there is some generally ac- 
cepted criteria for judging their worth and unless reports contain enough 
information to allow such judgments tc be made." 

Dr. Hoos is much more blunt. She claims that the use of systems 
analysis to solve social problems is an "opportunity for enormous boon- 
doggling ... by the very ascription of rationality , there are built-in 
assumptions about whose conception of rationality is reflected. Ratio- 
nality is not a dye vat into which all may be dipped and come out the 
same color. Rationality is neither absolute nor constant, but depends 
on the frame of reference. Rationality knows many times, places and 
forms. What could be economically rational could be irrational socially; 






231 



what could be socially rational could be political dynamite; what could 
be politically rational could be totally irrational from an engineering 
point of view." 

To cite examples of what can happen, Dr. Hoose points to Federal 
grants, to information experts, to design systems which diverted people 
from the task of delivering health services in some more efficient or 
effective way. "In many instances," she says, "it costs more to pro- 
cess information about a patient than to cure him. And more special- 
ists live off the systems set up to conquer certain diseases than die 
from those diseases." 

Well, at least there's not much doubt about where Dr. Hoos stands. 
She thinks we are wasting a lot of time designing systems, models, and 
games when we ought to be using our limited resources to solve problems 
with more obvious solutions. Her point about whose rationale is so 
relevant to the politicians that they have bad dreams about these things 
at night, I suspect. 

We all know what a pilot project is; it's a Government activity 
without enough money in it to accomplish very much. But ideally a pilot 
project is an attempt at something new. If, after evaluation the project 
pays off, then Government or the private sector at some level presumably 
expands the idea to whatever is the appropriate universe. An example 
of this kind of approach is the notion by the Department of Housing and 
Urban Development to give poor people money so that they can purchase 
their own housing, without the Government having to go through the busi- 
ness of providing the more familiar types of housing subsidies. HUD 



71-867 O - 76 - 16 



232 



hired a number of contractors to do elaborate forms of evaluation on how 
this might work out. 

However, in May, Dr. David J. Fitch, 54/ while conceding that Con- 
gress needs better information on the expected effects of such pilot 
programs, added two more problems to the list which "make it difficult 
to carry out and evaluate experimental programs . . . : 

1. The agencies which have responsibility believe that 
Congress will object to the random allocation of program bene- 
fits needed if effects of the programs are to be determined. 

2. Those within an agency who have responsibility for 
research design may be limited by their own competence and 
are almost certainly limited by those above them who feel a 
need to 'play it safe, ' and therefore will not support good 
evaluation research design because of anticipated public re- 
lations problems. 

My thought is that programs with regard to these dif- 
ficulties could be made through the establishment of a grant 
operation under the Senate Government Operations Committee 
in which proposals for evaluation research from the agencies 
would be considered by a staff and advisory group, and funds 
awarded to support good research — the philosophy being that 
the more Congress demonstrates it wants good evaluation, the 
better the evaluations will be. 

Ah, but Mr. Fitch, you haven't answered Dr. Hoos' problem about 
"whose rationale." 

The GAO auditor wants to be isolated so that he can draw his own 
"objective" conclusions — in other words, his rationale . Similarly, 
the National Science Foundation's RANN support of evaluations for a 
series of public concern will be apart from Government. Dr. Selma J. 
Mushkin of the Public Research Laboratory at Georgetown University 



54 / Research Psychologist Health Resources Administration, U.S. 
Public Health Service. 



233 



suggests, "and in the strength of its isolation from Government, it may 
also find it is removed from the realities of governments, or even in 
its isolation, produce an environment hostile to change." 

Dr. Mushkin shows some respect for the evaluation process at the 
Department of Defense, without endorsing it entirely: 

"It is a model that yields much dialogue, testing, research, re- 
assessment and multiple reviews prior to application. Yet it is of 
Government, produced both in house and out without the fanfare that has 
accompanied some civilian sector reports .... Surprisingly, there 
appears to be more scientific prodding and questioning in the military 
system than is currently built into the civilian sector evaluations." 

Dr. Mushkin points out that in the military, the evaluation pro- 
cesses built in at all levels in operations and budget from the Secre- 
tary of Defense to the military base, whereas multi-level evaluation 
in the civilian agencies is rare. 

She says, "evaluation and analysis are now so imbedded into the prac- 
tices of the military administrative structure that I think it would be 
difficult to sort out the analytical resources applied. In the civil- 
ian sector, each evaluation is a precious thing, or nearly so." 

Dr. Mushkin admits having no regular contact with the military eval- 
uative process. But it would only be fair to add to her praise that 
there are also dangers in the "everybody gets to stir and season the 
soup theory." Under McNamara particularly, weapons systems tended to 
become more cumbersome and complicated than necessary because each com- 
mand had something it wanted added. That's how you get individual and 
vulnerable war machines that cost millions of dollars per copy. 



234 



But the Importance of taking more than today's specific problems 
into account is made evident in a paper Dr. Mushkin gave in 1973 before 
the American Statistical Association. She pointed out that attempts 
toward policy analysis in the health field cannot be limited to health 
programs. 

She says, "Benefit assessments underscore the commonality of pro- 
gram purpsoes. Many public programs involve pricing human life, programs 
that are not directly associated with health care and that do not bear 
the label of health programs. The most familiar example is the highway 
construction program, where lifesaving costs or death and disability 
costs must be equated in deciding whether or not to construct a clover- 
leaf or add another highway lane." 

It gets complicated, and Congress knows how complicated it can be 
as evidenced by the remarks by Congresswoman Martha Griffiths at a hear- 
ing on "Problems in Administration of Public Welfare Programs" in June of 
1972: 

These programs affect over 60 million persons and cost 
$100 billion a year, and yet for years we have examined these 
programs one by one. But both the Congress and the executive 
branch are becoming increasingly aware of the fact that we can 
no longer afford to legislate and administer the programs in a 
piecemeal fashion when so many of the programs affect the same 
persons .... The purpose of these hearings to learn more 
about the rules and regulations under which programs are admin- 
istered at the local and state levels, and the actual adminis- 
trative procedures that are followed. 

Following the Congresswoman 's opening statement, the first witness 

was a casework aid, who said immediately, "I think the most important 

thing that I mentioned in the written statement, as far as dealing with 

our clients and the rules and regulations, is the fact that the grants 

are so low in relationship to the need that our clients have." 



235 



Don't groan. That's a legitimate statement. As a matter of fact, 
I don't believe Congress would give program evaluation as a device for 
policy formation a second thought if there was enough money to go around. 
But the members watch the Federal budget closely, and each year, the 
fixed costs eat up more and more spendable dollars, while inflation and 
interest on the debt chisel away at the other end of things. 

Many members of Congress are beginning to get the impression that 
many of the frustrations of the people with the Federal establishment 
stem from the fact that programs often do not operate as advertised and 
even if they do, there are too few dollars in each pot to solve the ori- 
ginal problem for all of the prospective users. A glaring example is 
the food stamp program which in just a few years has soared to be a $5 
billion program while at the same time advocates of the program complain 
only about 50 percent of the potential users are being served. 

As a result of situations like these, Congress wants the maximum 
program bang for the buck, and while it wants to know which programs work 
and which ones don't, the members clearly aren't too happy when the cat- 
egorical cripples are exposed in the daily press before there is time to 
make necessary orthopedic repairs. 

The Budget Control and Impoundment Act adopted this year will force 
many changes, the most basic being that under ideal operation, the De- 
fense Department will have to line up its priorities alongside those of 
the Departments of Health, Education and Welfare, and Agriculture so that 
Congress can decide which things are most important from a national stand- 
point to pour the dollars into. It is not too difficult to foresee one 
of two things happening: 



236 



Either the executive agencies will begin to do their own program 
evaluation to provide their authorizing committees with the ammunition 
to defend their operations before the budget committees, or all deci- 
sions will be made at the White House and cabinet officers will be rele- 
gated to housekeeping chores like second lieutenants at the Pentagon. 
If the latter selection is chosen by an Administration, there will un- 
doubtedly be many head-to-head confrontations between Congress and the 
White House. The winners in cases like this will be those programs with 
the strongest lobbies backing them up, or the annual glamour programs 
which the press enhances until it gets bored with them — you know what 
I mean: last year the environmental crisis, and this year the energy 
crisis. 

There may not be much room for appropriations by merit if this 
course is taken, and if recent history is a measure to judge by, pro- 
grams for rural people will probably suffer at the hands of an urban 
dominated House anxious to take care of its majority. 

Nevertheless — the new budget control act aside, Congress wants 
to know which programs work and which programs are not providing good 
public service given the dollars spent, and even though my instincts 
tell me to agree with Dr. Hoos when she says, "Systematic and analytic 
review of the systems techniques would reveal that they are not appro- 
priate to the myriad of social tasks to which they are being applied," 
I continue to be left with the frustration that I do not really know 
what is going on out "on the ground," as the foresters say, and there- 
fore I often find it difficult to provide the Senators I serve with the 
reliable information they deserve. I know further that one of the reasons 



237 



I don't know what is happening is because of intentional obfuscation by 
Federal agencies in many cases. Like the Job Corps survey mentioned 
earlier, they can make the numbers say anything they want — or anything 
that they think Congress wants to hear. 

When GAO came to see the staff of the Committee on Agriculture and 
Forestry following enactment of the Congressional reorganization bill, 
there was polite exchange of detailed letters between the Chairman, 
Senator Talmadge, and the Comptroller General. These sallies were fol- 
lowed by a round of meetings while the GAO tried to find out whether we 
wanted to dump ten computer years of work on them, and we tried to learn 
whether it would take a year to get a turnaround answer on a problem 
from GAO, as we felt was usually the case. 

These conversations went on for hours, gradually becoming more and 
more refined as the GAO staff began to understand our point of view on 
agriculture, rural development and forestry. It was only then that we 
could get down to the business of problem solving. 

So you see, in this case it was our rationale. 

Since these early meetings, the GAO has performed a number of worth- 
while services for the Committee — most of them rather homely tasks that 
have not required management information systems or computers. The GAO 
systems people worked with us in a somewhat more informal basis than had 
been their usual procedure, providing on-the-spot advice that was of more 
value than the written GAO report which gees through a final review pro- 
cess that takes two months or more. 



238 



As an aside here, it should be noted that the seven or more review 
steps plus executive agency review of a report is a scholarly way, and 
auditor's way of going about a study. Often, things move too quickly 
on Capitol Hill for that kind of work to be of much value. For instance, 
with the right kinds of pressures behind it, the committee can receive 
a bill, hold hearings, and have it reported to the floor in two weeks. 
But obviously there are long-term studies which are worth doing. The 
committee has a huge job underway right now. It involves a comprehen- 
sive review of the personnel of the Farmers Home Administration to mea- 
sure the competence, number, and job attitudes of the personnel matched 
up against the programs which Congress has given the agency to accom- 
plish. 

This study is interesting because Senator Dick Clark, who asked for 
the study, declined to call it an investigation. Further, Clark has en- 
listed and received the cooperation of the Administrator of Farmers Home 
Administration in assisting GAO with the work. Thus far, GAO and Farmers 
Home have been working at the job in an admirable manner. 

It is our hope that the completed study will draw rather broad sets 
of conclusions upon which the committee can base future decisions con- 
cerning the future of the agency, and those decisions can be made in 
such a way that in negotiating any changes in law or administrative regu- 
lations with the Farmers Home Administration, both the committee and FmHA 
will be working from the same sets of numbers and assumptions. 

Hopefully the results of this study will provide a vital tool to 
aid in policy formation. 



239 



Also with the help of the GAO, the committee, under the leadership 
of Senator Humphrey, has had enacted a forest and rangeland management 
bill which is designed to provide us with a new systematic approach to 
decision-making on the National Forest System. We believe that this 
bill could serve as a model for future legislation in other fields. 

For the past several years, the air of Washington has been polluted 
with a mean little debate over the management of the National Forests. 
The Forest Service, surrounded by assaults from conservation organiza- 
tions and the timber industry was under a constant legal and verbal 
cloud. The agency was losing the confidence of the Office of Management 
and Budget to cope with its problems, and there is even evidence that 
there was a dispute within OMB on how the National Forests should be 
managed. Because of inadequate appropriations, millions of acres of the 
forests needed reforestation or stand improvement. Because of drastic 
personnel cuts by the Administration — even in the face of orders to 
cut more trees — the Forest Service was incapable of managing any of 
its other responsibilities as well as it knows how. 

The problem as Senator Humphrey saw it, was to cut the trees the 
Nation had to have (lumber prices were skyrocketing) and still provide 
for wilderness, other forms of public recreation, protect watersheds, 
wildlife and generally maintain a sound environmental balance in the 
forests. 

Once again, skull sessions got the job done. The staff of the Com- 
mittee sat down with representatives of a number of conservation 



240 



organizations. At the same time, and the same table, we had representa- 
tives of the industry and occasionally personnel from the Forest Service. 

The result of those meetings was not a new policy for the National 
Forest System but a device by which policy can be set. 

Briefly, the bill provides that periodic assessments be made of all 
of the renewable resources in and on the National Forest System. Using 
this assessment, the President is to develop a Program of Work for ten- 
year increments that is amendable by Congress. After the program is 
adopted, the President is required each year to request such sums as are 
necessary to meet the goals of the program, or if he does not ask for 
the money, he must explain why. 

Finally — and this is where GAO helped us out — the bill provides 

for program evaluation which will aid the Congress in determining whether 

the Forest Service is getting the job done. The evaluation language 

reads like this : 

For the purpose of providing information that will aid 
Congress in its oversight responsibilities and approve the 
accountability of agency expenditures and activities, the 
Secretary (of Agriculture) shall prepare an annual report 
which evaluates the compenent elements of the program re- 
quired to be prepared . . . which shall be furnished to Con- 
gress at the time of submission of the annual fiscal budget, 
commencing with the third fiscal year after the enactment 
of this Act. 

These annual evaluation reports shall set forth pro- 
gress in implementing the program required to be prepared 
by Section 3 of this Act together with accomplishments of 
this program as they relate to the objectives of the as- 
sessment. Objectives should be set forth in qualitative and 
quantitative terms, and accomplishments should be reported 
accordingly. The report shall contain appropriate measure- 
ments of pertinent costs and benefits. The evaluation shall 
assess the balance between economic factors and environmen- 
tal quality factors. Program benefits shall include, but 



241 



not be limited to, environmental quality factors, such as 
esthetics, public access, wilflife habitat, recreational and 
wilderness use, and economic factors such as the excess of 
cost savings over the value of foregone benefits and the 
rate of return on renewable resources. 

The reports shall indicate plans for implementing cor- 
rective action and recommendations for new legislation, 
where warranted. 

The reports shall be structured for Congress in concise 
summary form with necessary detailed date in appendices. 

I don't know what that language does for my reader, but for me it 
sings. 

Similar evaluative efforts by GAO have assisted the committee in 
its extremely detailed oversight hearings in the field of rural devel- 
opment, an area in which the Nixon administration did not choose to be 
responsive. An unfortunate result of this was that our detailed, sys- 
tematic questioning before, during, and after oversight hearings kept 
the small staff of the Rural Development Service in USDA totally tied 
up for weeks, with no time to d£ any rural development, even if it had 
been permitted to do so by the administration. 

However, it would be well to add that Congress now has the Admin- 
istration's attention in this subject area, despite the Executive's de- 
sire for the whole business to go away. 

As Dr. Arthur Miller, the constitutional scholar at George Washing- 
ton University points out, ours is a Government that provides for a 
strong Executive. As he says, no other major nation puts so much power 
into the hands of one man. In the past, Congress has added to those pow- 
ers by receding from its own prerogatives. But I believe those days are 



242 



over. Congress has taken a number of steps such as passage of the Bud- 
get and Impoundment Act to regain some control over policy and spending. 
There's a new day coming, but it will not arrive next week, because 
change in a democracy is necessarily slow and evolutionary (unless the 
protein of revolution of some kind is combined in the admixture) . But 
the Congress is leery of "experts" who have often led the member down 
the garden path, and it is comfortable with the voice of home folks, 
who are not experts. 

By and large, the lawyers, economists and generalists who are 
the staff of Congress are more aware that the term ought to be "systems 
analyses" or "system analysis" (as opposed to systems analysis) than 
they are of what management information systems can do to help them 
serve their masters. Further we are very nervous about using machines 
to put people problems into neat cubbyholes. But one thing is certain - 
Congress and its staff have one basic complaint: 

Somehow we've got to find out what's happening in those 
damn agencies so we can make some decent decisions around here 
for a change. 

That basic frustration will lead Congress to do something, but 

again, since there are 535 individuals on board, God only knows what 

that something will be, and it may be 535 somethings which may or may 

not be subject to systematic evaluation. 



243 



DISCUSSION 

During the course of his research evaluating programs of the 
Department of Agriculture, Mr. Giltmier discovered that the agency 
refuses to release some evaluation reports to the Congress. In this 
same respect because of a lack of cooperation from the Department of 
Agriculture, it took, the General Accounting Office eight months to 
compile a list for the committee of evaluation reports done by or 
for the Department of Agriculture. 

Mr. Giltmier noted also that sometimes the General Accounting 
Office's assistance is inadequate for the committee's legislative 
oversight needs. One clear cause for this inadequacy is the "long 
turn-around time" for much GAO work. The Office employs a seven- 
step review process. As a result a year may elapse between the com- 
mittee's request for a project, its completion, the seven- step 
review and delivery. The committee may find these time-delayed re- 
ports inadequate because members are no longer interested in the is- 
sue. It would be helpful if the GAO could provide evaluation in- 
formation in a shorter time. 

Recently the General Accounting Office instituted two reporting 
procedures which circumvent the need for long reviews, thereby pro- 
viding the Congress with information on a more timely basis. These 
consist of (1) presenting requesting committees with oral briefings, 
or (2) transmitting informally reviewed draft reports, which 
committees sometimes use as final reports. 

Mr. Giltmier noted another potential problem: that the objec- 
tives of many GAO studies do not coincide with legislative infor- 



244 



mation and oversight needs. It was noted that seventy percent of 
GAO studies are self-initiated. However, it was also remarked that 
many of 'the GAO studies are done to fulfill statutory requirements 
made by the Congress. 

The point was also made that sometimes legislative actions 
themselves hamper effective evaluation. Several factors contribute 
to this dilemma. First, many different laws are enacted relating 
to similar types of programs and program delivery. Sometimes these 
overlap or are duplicative and sometimes they are contradictory. 
This happens especially in the agricultural area. The suggestion 
was made that before drafting new legislation, staff should spend 
more time determining if there already are laws relating to the is- 
sue, or if existing statutes merely require modification. 

The second problem, mentioned repeatedly throughout the seminar^ 
is the fact that legislative statements of program goals or objectives 
are imprecise or vague. This makes it difficult for evaluation re- 
searchers, including those in the GAO, to develop yardsticks of pro- 
gram goals against which to measure actual accomplishments. It is 
difficult to reconcile this problem because of the need for political 
compromise in writing and passing legislation. 



245 



DAY III: EXECUTIVE BRANCH EVALUATIONS AND THE PROCESSES 
OF EVALUATION: SUMMARY OF PROCCEDINGS 



A. Activities Underway In the Office of Management and Budget to Improve 
Evaluation Activities 

Clifford W. Graves was the lead-off speaker for the third day 
of the seminar, May 19. He began his presentation by tracing the evolu- 
tion of the Bureau of the Budget's and the Office of Management and 
Budget's concepts of evaluation. At first the Bureau of the Budget viewed 
evaluation exclusively in terms of efficiency. Subsequent attempts by 
the BOB and the OMB to view evaluation in terms of program performance 
and accomplishments, using the tools of the Planning, Programing, Budget- 
ing System and Management by Objectives, did little to meet decision- 
making needs. However, these efforts left an important legacy for the 
present, since they influence the mode of thinking of staff of OMB, GAO, 
and Federal agencies, and they provide a pool of people experienced in 
thinking in terms of program performance and output measurements. 

The OMB has conducted several assessments of the quality and use of 
Federally funded evaluation studies. The assessments, according to Mr. 
Graves, indicate that most of these evaluation efforts were not timely 
enough, inaccurate, and useless or irrelevant for decisionmaking needs. 
The OMB is now pursuing a vigorous effort to improve the conduct and 
use of Federally funded evaluations. The OMB Office emphasizes that evaluation 
should not be an academic exercise, but rather should be an integrali 
on-going part of Federal agency operations, effectively integrated into 
management, program planning, and preparation of budget justifications. 



246 



The OMB provides technical assistance and policy directives on 
evaluation to Federal agencies. In the process, however , the agency 
stresses that Federal agencies themselves, and not the OMB or the Con- 
gress, should be the focal point for improving the conduct and use of 
evaluations. 

OMB efforts to strengthen agency evaluation capabilities consist of 
encouraging agencies to use a "decision-driven" model for evaluation. 
This requires agencies to gauge where evaluation will be used in decision- 
making before funding evaluation research. At the same time the OMB is 
encouraging the Congress to include appropriate program evaluation re- 
quirements into legislation authorizing new programs. 

During 1974 the OMB prepared an inventory of Federal agency evaluation 
research activities and offices. It is conducting a joint project this year 
with the General Accounting Office to update this study and to gather 
more information about the funding of evaluation research. The OMB 
recently published a guide for managing evaluations in Federal agencies 
and has plans for the following publications: an assesment of the use 
of agency evaluations, an assessment of evaluation training needs, and 
a technical manual to assist agencies in structuring evaluation manage- 
ment systems. 

B. Activities Underway in the General Accounting Office to Improve 
Evaluation and Evaluation Research 

The Congressional Budget and Impoundment Control Act of 1974, P.L. 

93-344, gave the General Accounting Office several responsibilities to 

assist the Congress in improving program evaluation. Among these are the 

responsibility for providing assistance to requesting committees on evaluation 

methodology, helping committees to develop guidelines for legislative 



247 



language for evaluation, developing a capability for synthesizing evalua- 
tion reports, and cooperating with the Office of Management and Budget 
and relevant committees in developing program-related information systems. 

Keith Marvin of the General Accounting Office provided details 
about activities the agency is taking to implement these responsibilities. 
They include: 

- development for each agency of an inventory of evaluations completed, 
in process, or planned, including a short description of the agency's 

total evaluation plan; 

- development of an inventory for each agency, giving the title of 

an evaluation report, the title of the program evaluated, the date the 
legislation was authorized, and the date of the evaluation report; 

- preparation of abstracts of each evaluation report, identifying 
the source or performer; and 

- development of criteria and standards for congressional committees 
doing evaluations, to assist them in making tradeoffs between different 
types of evaluation strategies in relation to specific evaluation related 
decisionmaking needs. 

C. The Preparation of a Draft Manual on Evaluation for Legislative Users 
Harrison Fox is Research Director of the staff of Senator Bill Brock. 
William Foskett has served as a part-time consultant to the Senator. 
These two staff members have made important contributions to improving 
legislative staff capabilities for evaluation. For instance, under 
Senator Brock's authorization, Dr. Fox has held a monthly seminar for 
staff, bringing together legislative users of evaluation, executive branch 



248 



program managers and users of evaluation, and evaluation 
performers to discuss case studies, general methodology, and ways of 
enhancing the use of evaluation research in decisionmaking. Also, Drs. 
Fox and Foskett are co-authors of a draft Program Evaluation Manual for 
Legislators and Legislative Staffs . The manual gives general principles 
about evaluation in terms relevant to legislative staff who have responsi- 
bility for incorporating evaluation requirements into legislation. 

Dr. Fox began the joint presentation by describing the importance 
of evaluation as a tool of oversight. But he also noted that this tech- 
nique is only one of several essential oversight tools the Congress uses. 
He gave the following typology: 

Type of Oversight Methods Which Can be Used 

Technique 



Legislative 



Hearings and meetings 
Congressional veto 
Statutory controls 
Nonstatutory controls 
Reporting requirements 



Fiscal and Analytical 



The appropriations process 
Control through audit 
Program evaluation 
Achieving priorities 



Investigative 



Investigations 
Constituents Problems 



Authoritative 



Impeachment 
Confirmation 



249 



Dr. Foskett's presentation consisted of a short description of the 
manual and its uses on the basis of his several years of experience 
conducting evaluations in education and developmental psychology, he also 
commented on the conduct of evaluation. 

Like Dr. Townsend, Dr. Foskett noted that legislative staff must 
work very carefully in developing legislative language for evaluation. 
Staff do not necessarily have to be concerned with the specifics of an 
evaluation research design, but it is essential that they be familiar 
with: 

- the need to formulate correct legislative statements of program 
goals, capable of serving as yardsticks or achievements against which 
programs can be evaluated; 

- general methods of experimental, quasi-experimental, and non- 
experimental evaluation research designs, including their funding and 
time requirements, and the differences among types of evaluation in 
relation to legislative needs, in order to formulate legislative re- 
quirements for the right type of evaluation design in relation to the 
questions which need to be answered; and 

- the need for close communication between user and doer of evalua- 
tion to encourage the generation of information most relevant to decision- 
making needs. 

Legislative users must also be aware of the political constraints 
on evaluation, that is, be able to identify evaluation priorities in 
relation to political expediency, and to know when and why agencies 
might choose not to evaluate, or not to use the evaluation information 
which has been generated. 



250 



As another important point, legislative users who have a responsibility 
for writing legislative language for programs requiring evaluation 
should be aware that some types of quasi-experimental and experimental 
impact evaluations necessitate expensive longitudinal data collection 
procedures. 

Legislative users also must make tradeoffs on another issue, that 
is, can existing evaluations of these programs or syntheses of evaluations 
of the same or related programs serve the evaluation- related decision- 
making needs of the Congress. 

Utilizing a framework developed for the manual, Dr. Foskett then 
outlined the types of evaluation techniques and tools which could help 
generate information for twenty different types of legislative over- 
sight questions: 



251 



■tutkiu i* raiquu aica ht u to ma onuicrr qnrrion uits in is-ituum i 






mmu m ...»i.«i »«■ 



.,.. t.1^1 Ijcknl.* 



I. UUiwTfM rolrl.v.1; pilot tMU0«. 



Pupplp rmJKi! To proriao lnc« 

«'»f «i tYtfiT f<mi 



I. Miicy iuImii; 



pilot tutU|; pi — ll f II 111 polity •Ml/cl* 



»ll» — *4»»c7*t »1— . orogroa I 



U lo.lil.tl.ro htitftrj) 



if—" 1 ": ^ *-»" 






oooaot of mlookmt;; unlilutM of ro*al»».. f.. rn.la.iio. 



flui u-17.1. 



pro»rW.».l««tlo« 4**1... ojoji 






I".. tKo t ool. or nooo. 



;l.l«cloa orl.lnollr .date 



•ftovoh Co iolvo tho pto.1*. 



M.l».ll, .«,..... ., 



D. Issues Relating to the Methodology of Evaluation 

From his vantage point as a social scientist and the president of a 
large consulting firm which does a considerable number of evaluations for 
the government, Clark Abt presented a comprehensive statement on the 
state of the art of the methodology of evaluation and on issues surrounding 
the use of evaluation. 



252 



Although he devoted a considerable portion of his paper to the 
methodological limitations of evaluation, Dr. Abt noted that some kinds 
of evaluations are more methodologically sound than others. For instance, 
it is easier to do cost-ef fectiveaess evaluation than cost-benefit evalua- 
tion. In cost-effectiveness evaluation "some criterion of effectiveness 
is set, such as advance in reading scores in an educational program, and 

the amount of advance per unit of cost is the cost-effectiveness It's 

basically a measure of the desired program output over cost input." 

"Benefit-cost is harder to compute because here we are computing 
benefits and costs in commensurate, usually dollar terms. We're trying 
to estimate what amounts to computation of return on investment, so that 
we can compare different programs. It is hard to compare years of read- 
ing achievement per dollar, with percentage reduction in cancer fatali- 
ties per dollar two completely different programs in education and 

health. If you have benefit-cost ratios for both, or return on fungible 
resources invested, you can compare those and when you can compare across 
programs ideally, you can optimize the whole protfolio of government 
investments by what the economists call marginal utilities. 

However, the major obstacle to effective evaluation, according to 
Dr. Abt, is the fact that it is not used by decisionmakers. Several 
factors account for this situation. One is that some decisionmakers 
expect more of evaluation than it can deliver. For example , he noted: 
"What we cannot do with program evaluation is create new programs, make 
decisions, make the assumption of rationality come true, and get users 
or consumers of program evaluations to ask the right questions. We also 
cannot make value judgments that have any credibility where these are 
decisive program decisions." 



253 



He also noted that communications gaps besiege both doers 
and users. Legislation does not always specify properly the objectives 
and potential decisionmaking needs of the evaluation study. This situa- 
tion could be improved if evaluation were conceived of as a "decision- 
driven" technique to be used in specific legislative oversight functions, 
such as budgeting, program formulation, or management. Related to this 
point is the abrogation of responsibility by the doers of evaluation, 
who are not sensitive to the political needs or connotation of evaluation 
information and make little effort to aid decisionmakers in assessing 
the policy implications of evaluation. 

Abt also stated that the users of evaluation are not always 
aware of the possibilities and limitations of evaluation research. As a 
result users frequently do not ask the right questions of evaluation or 
do not receive adequate payoff for the dollars invested in evaluation 
research. Abt noted, for instance, that never once in his twenty 
year career as an evaluation researcher had legislative users of evalua- 
tion consulted with him regarding the objectives, methodology, or possible 
use of an evaluation study. 

Dr. Abt emphasized that while the technology of evaluation research 
had improved considerably in recent years there have been very few 
theoretical breakthroughs in evaluation research methodology or concepts. 
Both evaluators and decisionmakers have a responsibility to improve this 
situation. They can do this by: 

- requiring evaluations only of the most important priority programs, 
not of every program legislated; 

- doing cross-cutting evaluations, which look at the objectives, 
effectiveness and impacts of programs with similar goals, authorized by 
different statutes, and administered by different Federal or state agencie 



254 



- requiring Federal evaluation managers to write better RFPs and 
modifying procurement regulations so that technically non-competitive 
firms do not waste inordinate amounts of time and money writing pro- 
posals, and 

- sensitizing decisionmakers to make decisions (and to accept the 
consequence^ about when "quickie" evaluations are required and when 
longer, experimental evaluations with randomized test and control groups 
are needed. 

Among the other weaknesses Dr. Abt identified were: 

- inadequate Federal support for training evaluatbrs in the use 
of statistical methods of analysis; 

- insensitivity to the selection of indicators of effectiveness 
and absence of well-formulated relationships between measures of effec- 
tiveness and possible policy options; 

- inadequate review by evaluators of the literature which already 
exists on evaluation; and 

- insufficient awareness by both evaluatorB and decisionmakers 
about the possible effects of an evaluation or an experiment on the 
participants of the program. 

E. Evaluation in the Office of Education 

Dr. John Evans, Assistant Commissioner of Education in the Office of 
Education, described issues relating generally to federal procure- 
ment and use of evaluation, using the Office of Education as an illustra- 
tion. 

He began by describing several problems common to most Federal 
agency evaluations. He noted, for instance, that some evaluation manage- 
ment systems are very disorganized; many RFPs are poorly written, and 



255 



frequently the academics who do evaluation research for the Government 
orient it more to support their basic research or training needs than 
to serve governmental information and analysis needs. He noted that 
the most effective evaluations are sponsored by agencies which have 
special program evaluation offices, with a technically competent staff, 
linked closely to the agency's decisionmaking center. 

There is a special program evaluation office in the Office of 
Education. It has responsibility for evaluation, planning, and budgeting. 
The Education office has a staff of 20 people and a budget of $13 million tc 
evaluate Federal education programs costing $7 billion. 

The following procedures are used to do evaluations in the Office 
of Education. The evaluation staff develops a list of priority evalua- 
tions which can be funded from the basic evaluation research budget. 
Agency staff familiar with evaluation research and social research 
methods draw up a technically sophisticated evaluation design before 
issuing RFPs to avert the problem of proposal preparation and sub- 
mission by technically unqualified performers. After proposals are 
submitted, an intra-agency technical committee reviews proposals, 
selects the contractor, and closely monitors the research to ensure that 
it meets the agency's needs. Monitors pay especially close attention 
to the methodology used in the evaluation study, since frequently evalua- 
tions can be discredited for faulty methodology. 

After the evaluation study is completed, the agency project monitor 
prepares a memo summarizing the evaluation report and identifying any 
policy implications which might have been generated. The memo is sent, 
via the Commissioner of Education, to the Office of Management and Budget. 



256 



F. Concluding Comments on "Evaluation Evaluation" 

Allen Schick's presentation on "evaluating evaluation" was a 
succinct commentary on some of the most important themes which had been 
raised in previous sessions. He noted that although Federal program 
evaluation budgets have continued to increase, program evaluation suffers 
from deficiencies of conceptualization, methodology, and use in policy- 
making. 

Dr. Schick said that agencies and the Congress are confronted with 
numerous problems in doing and using evaluations. First, it may be 
advisable for agencies to separate evaluation from budgeting, since 
agencies usually try to maximize their budgets and retain programs. 
Second, because of its critical adversary nature, evaluation might be 
performed best in a separate inspector general office. Third, agencies 
tend to avoid objective evaluation because they are the captive of 
their programs and would like to believe that programs are, indeed, 
benefitting their clients. Fourth, evaluation is very difficult to do. 
Data collected in the normal course of managing a program are insufficient. 
It is exceedingly difficult to collect and interpret hard-to-measure 
information on program impacts. Fifth, administrators as well as members 
of Congress may sometimes shy away from using the results of evaluations 
since evaluation often indicates that programs are ineffective. 
This problem is compounded since most evaluations are done on innova- 
tive social programs, not on long-established programs which may profit 
more from good evaluation. Sixth, at times evaluation of new programs 
takes precedence over evaluation of past enactments. This is all the 
more significant because committees can become more "program advocates" 
than program overseers. Therefore congressional oversight may suffer. 



257 



Dr. Schick remarked that the Congress is now gearing up to perform 
a better oversight role. However, he contended that the Congress should not 
as a rule get directly into the business of conducting evaluations. Congress 
should be concerned with legislative oversight; this is not analogous to 
program oversight, which should be the responsibility primarily of executive 
branch agencies. 

The Congress can assist in improving legislative oversight and the 
use of program evaluation, Dr. Schick said, by taking the following steps: 

1. Improve the identification of goals and meaning of program 
objectives and congressional intent in legislation mandating 
new programs ; 

2. insist that evaluations be kept honest and that impacts, not 
only process issues, be measured; 

3. permit more time for the conduct of complex longitudinal 
evaluations of ongoing programs; 

4. insist that evaluations be concise and readable to the layman; 
and 

5. make greater efforts to use evaluation information in the 
authorization and appropriations process. 



258 



"FEDERAL PROGRAM EVALUATION: THE PERSPECTIVE FROM OMB," A PAPER 
PREPARED FOR THE AMERICAN POLITICAL SCIENCE ASSOCIATION BY CLIFFORD 
W. GRAVES AND STEFAN A. HALPER, USED AS A BASIS FOR REMARKS PRESENTED 
BY CLIFFORD W. GRAVES, OFFICE OF MANAGEMENT AND BUDGET 



This paper discusses the principal issues relating to governmental 
evaluation activities and traces the evolution of OMB's response to them 
since the 1950' s. It concludes with a description of our current approach 
and some challenges to the professional and academic evaluation community. 

The main thesis of the paper is that the definition and role of eval- 
uation in which governmental activity is conducted, to label various ap- 
proaches, successes or failures whithout relating them to their contemporary 
context is misleading. 

A secondary thesis is that there has been no revolution in governmental 
evaluation; "evolution" is a more appropriate description. Current practices 
build on the lessons of the past, and future approaches will follow the 
same pattern. 

Finally, the authors are concerned about the ability of the evaluation 
profession to respond to the current challenge. There is a noticeable 
divergence between the demands of the governmental decisionmaking process 
and the techniques and processes of the evaluator. 



*First delivered at the Midwest Political Science Association meeting, 
Chicago, Illinois, May 1, 1975. Remarks delivered were based, in part, 
on this paper. 



259 



The Importance of Program Evaluation 

In the current period of budgetary stringencies, government decision- 
makers are continually faced with perplexing and diffcult options. The 
dollar purchases fewer goods and services today than in past years, and 
resources for new programs must be secured at the expense of existing ones. 
More than ever before, decisions concerning programmatic priorities should 
be based upon as broad and objective an information base as possible. Program 
evaluation is a process that enhances objective decisionmaking; it provides 
a unique input to decisions bearing on effective program management, the 
relative effectiveness of alternative approaches, and the substantive im- 
pact of government activities. 

Given the rising demands on government and shrinking resources avail- 
able to it, the role of evaluation has become critical. Both as a manage- 
ment tool and a measure of impact, evaluation may come closer than any 
other process to providing the objective information central to decision- 
making. 

Working Definition 

Like "planning" and 'coordination," "evaluation" is a loosely used 
term in governmental circles; it lacks any universally accepted definition. 
Like the other two terms, evaluation is very difficult to oppose in prin- 
ciple, even when one is not sure what it is. For the purpose of OMB 's 
mission (and this is applicable to most of the domestic Federal agencies) 
program evaluation can be defined as a management process that seeks to 
systematically analyze Federal programs (or their components) to determine 
the extent to which they have achieved (or are achieving) their objectives . 
The essential characteristic which distinguishes evaluation from other forms 



260 



of research is intended use : It is decision-oriented. 

Issues 

Three interrelated issues perennially confront 0MB and others concerned 
with governmental evaluation policy and practices: 

— selection and design of evaluation projects; 

— utilization of evaluation findings; 
"evaluability" of governmental activities. 

They will serve as reference points for the discussion which follows: 

Projects Selection and Design . The planning of evaluation programs — 
the process of determining what to evaluate, when to evaluate it, and how 
to go about conducting the evaluation — is one of the less-developed areas 
of public management. Yet, this fundamental first step in the evaluation 
process can make or break project success. A 1974 0MB survey of current 
agency practices revealed a wide variety of approaches; we suspect that 
this variety has changed little from the situation 15 years ago, when the 
first attempts at more rational planning were undertaken. 

Use of Evaluation Findings . No professional likes to see his work 
gathering dust on a bookshelf. Yet, this fate befalls many evaluation 
project reports of both high and low technical quality. At 0MB, this is 
disturbing both from a budgetary and program management standpoint. 

The number of potential users of evaluation findings is growing. 
The governmental decisionmaking process is more pluralistic and open 
than even before, and the demand for objective information has also 
increased-. The issue of who receives evaluation findings and how the 
findings are used, therefore, looms very large among our concerns. 



261 



Evaluability 

Wholey and others argue that much if not most of what the Federal Govern- 
ment does is not "evaluable;" i.e., the rigorous analysis called for in our 
definition is either impossible or yields no new useful information. 55/Whether 
or not one fully supports that conclusion, the fact remains that many govern- 
mental activities have no basis in testable theory, lack measurable objectives, 
or exist beyond effective management control. 

This is not a new phenomenon, indeed, the newer programs initiated by 
the Congress and the Administration are much better designed than older ones 
were. However, the issue of how to design and manage a program so that 
evaluation is possible is far from being resolved. 

Evolution of the OMB Approach 

In reviewing the executive branch and, more specifically, OMB's 
interest in program evaluation, two significant trends stand out. The 
first of these is the changing focus of evaluation, from efficiency in the 
1950's; to statistical program output in the 1960's; to a more generalized 
program performance measure in the 1970' s. 

At the same time, OIlB's approach to evaluation move from one of no 
systematic interest in the 1950 's to an internally focused system during 
the Planning Program and Budgeting System (PPBS) era, and currently to an 
external or agency focus in the early 1970' s. 



55 / Dr. Joseph S. Wholey, Staff member of the Urban Institute, Washington, 
D.C. 



262 



Background (prior to 1965) 

During the Eisenhower Administration, BOB was primarily concerned with 
operational efficiency. This reflected the heavy emphasis within BOB on 
numbers rather than program substance and policy. To a large extent, there 
was no such thing as formalized program evaluation. The budget examiners 
did special analysis, had frequent contact with the agencies, and further, 
the Federal budget was structured along organizational lines, rather than 
programmatic lines. 

In the early 1960 's under the Kennedy Administration, the focus of the 
budget examiners shifted toward the substance of programs. However, 0MB 's 
interest in program evaluation per se remained unstructured. With the 
Great Society and the creation of large numbers of new programs, 0MB 's 
operational structure became overburdened. At the same time, with the 
growth of a new professionalism in the public administration field, new 
forms of relating expenditures to effects became known. This trend was 
largely responsible for the development of the PPBS in 1965. 

The PPBS ERA (1965-1970) 

The Planning Program and Budgeting System, first operationalized in 
the Department of Defense, was primarily directed at improving the overall 
management of Federal Government resources. It attempted to link program 
funding decisions with program actions and assessments of benefits. The 
key elements of the PPBS were: 

— a program structure which organized agency activities in terms 
of objectives or output and costs; 

— program memoranda which contained decisions on current major 
issues and the rationale for those decisions, including of options; 



263 



— specific analytic studies which analyzed the issues; and 

— program and financial plans which presented in tabular form future 
cost implications of major program decisions. 

Reduced to its essentials, the PPBS was an attempt to provide for 

a more orderly and reliable decision process by clarifying and making more 

explicit the reasons for decisions. 

Most important, PPBS represented the concept that government carries a 
responsibility for achieving specific objectives rather than simply trying to 
make general improvements in a particular area . This change of orientation 
implies a close examination of program impact as a part of the public management 
process. 

From the perspective of program evaluation, PPBS represented the first 
significant systematic approach to the development and use of program evalu- 
ation as a part of the public decisionmaking process. Program evaluations 
were directed toward issues which were cross-cutting in nature; the evaluations 
were designed in the form of specific analytic studies. Program memoranda 
were then developed which used evaluation information in policy analysis and 
decision options. The primary users of PPBS evaluation information were OMB 
and Administration policymakers, rather than agency program managers and the 
Congress. 

The system was not specifically decision-driven except to the extent that 
evaluative studies were undertaken in priority issue areas. There was no 
established process for structuring programs in a manner which would neces- 
sarily render them evaluable. 



71-867 O - 76 



264 



Observations About the PPBS ERA 

Many people have speculated on the reasons for the demise of PPBS. The 
precise reasons for its decline are not relevant to us here. However, what 
is relevant is the attempt to deal with the Federal management process and 
the integral nature of evaluative activity within that process. Many of the 
underlying premises of PPBS are taken as givens today; for example, the focus 
on program and program impacts. PPBS also contributed much to the development 
of professional skills within government to evaluate and judge program per- 
formance. Program evaluation and measurement techniques, even today, are 
not perfect; but it should be recognized that PPBS was the first Government- 
wide management system to focus on improving evaluation procedures and in- 
tegrating evaluation information into the decision process. We are still 
learning. 

The Performance Management System (1970-1972) 

As the limitations of PPBS became clear, BOB initiated a study to 
investigate possible new approaches. The focus was the possible inte- 
gration of the PPB and the then-existing budgetary system. 

The study concluded that integration was possible. A system was 
devised which measured program effectiveness in terms of impact vs. bud- 
geted resources consumed. It also provided for the timely reporting of 
important information to decisionmakers. Thus, the basic structure of the 
Performance Management System was developed. In late 1970, the President 
authorized 0MB to proceed to test the system by implementing it in selected 
high priority domestic programs. 



265 



The PMS concept was based upon the principles of management by ob- 
jectives and accountability for results. It recognized that the system 
had to be flexible enough to adapt to the variety of program and organi- 
zational characteristics found in the Federal Government. The intent was 
to develop and implement a system based on a broad concept without prescribed 
rigid formats for its operation. Essentially, the system concepts were: 

— clearly defined program objectives; 

— a designated manager who could be held responsible and accountable; 

— identification of performance measures with time-phased targets 
for performance; 

— periodic reporting of performance based on the performance mea- 
sures and targets; and 

— identification of significant variations in performance to allow 
for timely corrective action. 

The PMS concept placed responsibility on the program manager for se- 
lecting and designing evaluation studies. Information was to be integrated 
by him into decisions taken in reaching his objectives. Both process and 
impact evaluation were contemplated, although the concept of evaluability 
did not play a specific part in program design. 

The Performance Management System never moved beyond the pilot test 
phase. With changes in OMB 's top management, PMS was reconstituted with 
a narrower focus in the form of the Management by Objectives process (MBO) . 

Management by Objectives can be characterized as a limited management 
system for Presidential and Cabinet-level concerns. The MBO process, as 
it was actually instituted, attempted to focus attention on a limited 
number of specific Presidential objectives. Agency heads were encouraged 
to establish their own MBO systems for internal management. 



266 



However, in its original conception, MBO represented what could be 
characterized as another expression of essentially the same interests and 
objectives as those which gave rise to PPBS: the public management process 
used by the Federal Government. While it was less ambitious and less com- 
prehensive than PPBS, it was more pragmatic in that it did not attempt to 
deal with all of the activities of the Federal Government at once, only 
those that were of particular interest to the President and other top policy 
officers. 

The MBO process provides for the establishment of objectives and 
the periodic assessment of performance against those objectives. It did 
not attempt to relate those objectives specifically to the budget process, 
nor to any programmatic or organizational structure. It instead focused on 
the achievement of the objectives by whatever programmatic, organizational 
other means were available and appropriate in each case. 

The periodic assessment of performance of the objectives does not 
represent, to a limited extent, an evaluation component. However, there 
was an apparent conscious decision not to attempt to make evaluation of 
performance an integral part of the OMB role in the MBO process, except 
to monitor that performance. 

For the most part, evaluations have been initiated by agency pro- 
gram managers or OMB Budget Examiners to obtain a closer look at programs 
which may form a part of an MBO objective. Evaluations have been systemati- 
cally utilized to the extent that they relate to better program management. 
Like the PPBS and PMS, there has been no established procedure for ensuring 
program evaluability . The MBO concept, as it was actually implemented, fell 



267 



short of its goals in much the same way as PPBS but for different reasons. 
The MBO process failed to establish the management system necessary to sys- 
tematically develop and evaluate performance on the objectives. 

Refocusing Evaluation 

The reassessment of OMB 's interest in program evaluation concluded that 

future OMB activities should: 

focus on strengthening the agencies capabilities, rather than those 
of OMB itself, and 

— attempt to improve the quality and use of evaluation information in 
the decisionmaking processes without requiring it. 

In short, the approach should be to encourage the agencies to improve 
evaluation activities but not to mandate action nor dictate what improvements 
should be made. Some would call this consciousness raising. 

This conclusion — that OMB's focus should be on improving evaluation 
within the agencies rather than within OMB itself — is based on the premise 
that improvement in the quality and management of evaluation in the agencies 
will contribute better information to the decisionmaking process within the 
agencies, as well as within OMB. It recognizes that primary responsibility 
for program management, of which evaluation is a part, rests with the agency 
head. 

Current Thinking 

When the evaluation policy function in OMB was reconstituted in early 
1974, its objectives and operating style were based on both the lessons of 
the past and the changing governmental decisionmaking environment. Writing 
in Public Administration Review, Lewis and Zarb set forth the definition of 



evaluation used in this paper, and noted that: 

. . .OMB takes the position that each agency head is 
primarily responsible for managing the programs under 
his jurisdiction and that program evaluation consti- 
tutes an essential element of effective management. . . 
each Federal agency should be responsible for carrying 
out its own evaluation program. 

However, concern has been expressed as to whether the 
results obtained from many of these efforts have been 
as useful and relevant as they might be to major policy 
decisions confronted by program managers, agency heads, 
the President and Congress. 

Therefore, OMB will initiate a multi-agency cooperative 
effort to upgrade and strengthen the overall quality and 
relevance of evaluation carried out by the various Federal 
agencies and to assure that evaluation results are utilized 
in arriving at major policy decisions. 56 / 

To implement this approach, OMB now works with the departments and 
agencies collectively and individually, to improve their evaluation programs 
without imposing a singular management or evaluation system; and with the 
legislative branch on program design, information sharing, and similar eval- 
uation-related issues of common concern. 

Out of this collegial approach to evaluation policy have come a number 
of concepts that relate to the issues discussed earlier in this paper: 

— a decision-driven model for evaluation planning; and 

— an evaluation component for new Federal programs. 



Sft / Lewis, Frank L. and Frank G. Zarb, (Office of Management and Budget), 
Federal program evaluation from the OMB Perspective. Public Adminis- 
tration Review, v. 34, no. 4, July /August 1974: 312. 



269 



Decision-Driven Model 

A program evaluation project can be considered useful if it is timely, 
accurate, and relevant, i.e., it is available to a decisionmaker at a time 
when a decision is to be made, it is methodologically sound, and contains 
reliable information relevant to the decisionmakers' needs. 

Based on information from a recent OMB survey of Federal evaluation 
activities, we identified three prevailing approaches used by most agencies 
to plan evaluation projects. Each of the commonly used approaches, i.e., 
(1) projects undertaken on direct demand, (2) projects undertaken on the 
basis of a cyclical review of programs, and (3) projects undertaken as a 
result of attempts to conceptualize information needs, have one common 
characteristic — they rarely involve explicit consideration of the decisions 
they should be designed to support or identification of involved decision- 
makers . Consequently, utilization of the studies in arriving at program de- 
cisions is often fortuitous. 

Evaluation planners must focus on major decision alternatives or 
subject areas around which decisions are anticipated when developing pro- 
posed evaluation projects. Further, decisionmakers, program managers, and 
other staff units must review these proposed projects and approve or 
disapprove the evaluation plan based on the sufficiency and timeliness of 
information to be available for their individual decision needs. This 
approach, which we have called the decision-driven model, has several ad- 
vantages: 

— it provides greater clarity in the objectives of individual 
projects; and 

— decisionmakers are better prepared to receive and use evaluation 
findings for specific decisions. 



270 



Operationally, the model is difficult because of the discipline re- 
quired of both decisionmakers and evaluators, and because of the multiple 
decisionmaker mode that characterizes the Federal Government. However, 
these problems do not negate the value of the approach. 

This model not only guides project selection and timing, it also 
should influence project design. The central criticism with this approach 
is that decisionmakers, even in the best of circumstances, rarely project 
their evaluation information needs beyond a 6 or 9 month time frame; and most 
extant evaluation techniques cannot yield reliable information in so short a 
time. Therefore, with the exception of those cyclical decision processes 
like the budget, where decisions can be anticipated a year or more in 
advance, serious questions arise as to the usefulness of the studies. 

Since most decisions will be made without adequate evaluation infor- 
mation, and since the public and the Congress demand accountability for 
government spending and decisionmaking, it is critical that the academic 
and research communities develop reliable and accurate rapid turnaround 
techniques. The limited progress which has been made in this area falls 
into two categories. Preassessment techniques tell decisionmakers whether 
their information needs can be met through evaluation; this procedure 
offers indirect performance data. A second approach, based on the military 
quick reaction capability utilizes quick survey procedures and panels of 
experts to provide gross policy direction — e.g., terminate/reauthorize, 
increase/decrease program resources. We are suggesting that there is a 
pressing need for additional methodological research to address this problem. 



271 



Creative talent is needed in this area, for breakthroughs here can affect 
the entire spectrum of government decisionmaking. 

Evaluation Components in New Programs 

The development of adequate rapid feedback procedures represents, in 
effect, a tactical step in providing decisionmakers with generally reliable 
interim information. On a broader scale, it is OMB's objective to include 
appropriate evaluation components into programs at their inception through 
both legislative and administrative means. This will make evaluation easier 
and more useful later. 

On the legislative side, working relationships have been developed with 
legislators and the key committee staffs in both Houses to gain agreement on 
the nature, structure and actual legislative language to be used in bills 
so that information gathering, as part of the evaluation process, is ini- 
tiated as soon as programs begin. An OMB staff paper now being reviewed 
proposes that six elements be taken into consideration when drafting new 
legislation. 

1. State program objectives in a measure form. 

2. Seek agreement between legislators and administrators on the 
criteria that will be used (a) to measure the degree to which 
programs reach their objective and (b) to measure the degree 
to which programs impact the problem. 

3. Ensure that extant methodology is capable of generating the 
information program managers required for decision. 

4. Ensure adequate staff and funds are available to perform the 
evaluation. 

5. Ensure that there are clear provisions for the flow of evaluation 
information from evaluation to decisionmaker. 

6. Establish who will use the evaluation information. 



272 



On the administration side, the paper proposes that OMB instruct 
the agencies to include program objectives statements and evaluation plans 
for new programs or proposed legislative initiatives. 

The review of proposed legislation in OMB is currently channeled 
through the Legislative Reference Division to the appropriate program 
divisions. There, a budget examiner or management associate review pro- 
posals for compliance with certain statutory and Executive Order require- 
ments. Civil Rights, environmental impact, and cost analysis are three 
such requirements. The OMB circular would add the evaluation plan to this 
list. 

Much of the impetus for the proposal to build evaluation into new 
programs and other current policy initiatives stems from a survey of 
Federal evaluation activities performed by OMB in the spring of 1974. The 
survey was, in effect, a first step in understanding and strengthening 
evaluation activity in the major domestic agencies. It has provided baseline 
data on the nature and scope of current agency practices. The survey: 

— identified organizational settings and priorities of units 
for which program evaluation represents a major and continuing 
responsibility; 

— discussed the nature of operating relationships between evaluation 
units and units involved in related information producing activities. 

— described the processes employed to plan, coordinate and disseminate 
evaluation studies; and 

— obtained expenditure and resource information. 

An update is now in progress; this time the survey is being jointly 
conducted by OMB and the General Accounting Office. With the help of a 
panel of senior evaluation officials (generally the Assistant Secretary 
level) we have sought to relate the survey finding to the management issues 



273 



surrounding the decision-driven model described earlier. 

We have augmented this process through symposia and conferences where 
we have had a valuable exchange of ideas with the academic and research com- 
munities on the use of evaluation in public policy formulation. 

Future analytic efforts are planned on evaluation and the budget process 
and a review of evaluation methodology in selected subject areas. The papers 
are intended as a mechanism for eliciting throughtful comment on the issues 
and as reference points for OMB initiatives. For example, specific action steps 
identified from the current papers are: 

— An OMB Circular to provide general guidelines for managing 
evaluation activities and building evaluation components into 
new programs (now drafting). 

— In depth assessments of selected agency evaluation activities 
(to begin in July 1975). 

— A comprehensive analysis of the evaluation training needs of 
Federal executives, program managers, and evaluation staffs 
(to begin June 1975). 

— On a very limited basis, some technical assistance has been pro- 
vided to agencies (on request) in structuring evaluation systems. 

— We are providing some technical assistance to the OMB program 
divisions on evaluation projects and legislative issues, on re- 
quest. Labor's evaluation of the CETA program is an example. 

Conclusion A 

The current OMB effort has been structured so that, over time, it 
should result in a significant improvement in the contribution of evaluation 
to the policymaking process. 

While our primary responsibility is to improve the quality of evaluation 
information used by the President and by OMB, the approach I have described 
will benefit the Congress and the growing numbers of decisionmakers in 



274 



State and local governments. The response to the approach so far has 
been encouraging. 

The professional evaluation community will find their work subjected 
to more scrutiny from Federal sponsors than in the past. Agency managers 
are becoming more sophisticated in the use of evaluation, Congress, GAO, 
OMB, and others will be increasing their demand for timely and reliable 
evaluation information. 

As noted earlier, some better techniques are needed to satisfy this 
demand. More important, the professional community will have to make sure 
their products can stand the - test of implementation. 



275 



DISCUSSION 

In response to questions raised during the discussion, Mr. Graves 
reiterated that OMB and agency budget officers do not always use eval- 
uation information in planning and programming. This happens because 
information is not always timely or adequate. 

Several clarifications were made about separating the evaluation 
functions of setting priorities, conducting evaluations, and using 
evaluation information. It was noted, for instance, that there is a 
separate office of planning in the Department of Health, Education, 
and Welfare, which recommends priorities for evaluation studies to 
be conducted by different offices. However, recommendations for 
evaluations also filter up from the line agencies to the planning of- 
fice. 

One problem is common to all executive agency evaluations. This 
is the absence of a central reporting mechanism to collect and dis- 
tribute evaluation reports. According to Mr. Graves, agencies should 
devote more attention to developing evaluation libraries or informa- 
tion dissemination mechanisms. 

Mr. Graves also remarked that it would be advantageous to en- 
courage Federal agencies to make all evaluation studies public. This 
would enable critics to double check the work of evaluation researchers 
with respect to assumptions and methodology used, data gathered, and 
conclusions reached. 

A member of the audience raised a very important point tangential- 
ly related to the seminar theme. Do the OMB and Federal agencies use 
policy analysis? Mr. Graves replied that with the exception of the 



276 



Defense Department executive agencies do not make much use of policy 
analysis which incorporates the findings of program evaluations. Sev- 
eral social policy agencies have attempted to use policy analysis for 
planning and budgeting purposes, but these attempts have been futile. 
The final issue raised after the OMB presentation related to how 
the OMB might develop statements or objectives of program goals or 
objectives for use in evaluation. According to Mr. Graves agencies 
and the OMB use four information sources: 

- legislative language, 

- legislative history, 

- consultations with committee staff, and 

- discussions with White House and agency officials. 

In conclusion, Mr. Graves remarked that OMB has mixed reactions 
to legislatively mandated statutory requirements for program evalua- 
tions. Some, he said, are too stringent and impossible to fulfill. 



277 



'CURRENT EVALUATION ACTIVITIES IN THE GENERAL ACCOUNTING OFFICE,' 
KEITH E. MARVIN, GENERAL ACCOUNTING OFFICE 



My presentation will cover program evaluation activities which in- 
clude evaluation methods, evaluation synthesis, and evaluation and mod- 
eling sources. First, with regard to methods, we are developing stan- 
dards and guidelines for program evaluation and doing other work closely 
related to what Harrison Fox will be talking about later. Our synthesis 
activity is where we will be putting together and refining evaluation 
information related to specific program areas or issues. Incidentally, 
I don't want to leave the impression that all of the GAO work on pro- 
gram evaluation is done in this staff. All GAO divisions are involved 
in program evaluation and the handout I have provided to you lists about 
60 reports issued in 1973 and 1974 which contain or deal with evaluation 
of Federal programs. (See Appendix ,J One of the handouts describes 
9 of these reports in more detail showing the kind of constructive crit- 
icisms which GAO has made to make program evaluation more valuable in 
decisionmaking in a variety of Federal programs. (See Appendix*) 

Basic Evaluation Information 

Our major activity since January 1975 has been to develop and con- 
duct jointly with the Office of Management and Budget an inquiry into 
department and agency evaluation activities. As chart 1 indicates, we 
are collecting information on recently completed evaluations, on in- 
process evaluations and on the plans at least by program area for eval- 
uation to be conducted in the next fiscal year. As the chart also in- 
dicates, we also cover the evaluation process in the inquiry and we in 
GAO will be interested in whether those processes are working properly. 



278 






\ 3 



: 









ۥ> 








CO 








UJ 


CD 






« 


!UJ 






CD 


£-«=- 






£*£ 


Livj 






P. 


—j 








Pp. 






*■•* . -wt 


rr ••« 






C3 


s -» 






HI qo 


O 






F— 


C3 


tn 




IS " 


>- 


CO 


£51 


, « 


-■^ 


C3 


r i 


•■'a 


» 


r « 


r. -m 


Lj 


' 


P». 


*c 


>- 


• i 


■r-?~-5P 


-~*J 


C.,3 


: : 


iwi--c» 


cs» 




1 


I 


I 





o 



279 



However, if the program described by Mr. Graves works out well perhaps 
we will not need to spend as much time worrying about whether the pro- 
cess is working as in providing better information to the Congress. 

The kind of information that we will be able to provide to com- 
mittees is illustrated by the loose leaf book which I have here today 
covering the evaluation information obtained on HEW and Department of 
Labor programs. The booklet is organized into sections for the major 
areas such as education and health. 

Examples of Evaluation Information 

Chart 2 is an illustration of the kind of information entered in 
the booklet which I just described. On the right is the title of an 
evaluation report and the date of issue. This is associated with the 
title of the program evaluated in the center column and on the left is 
shown the legislation which authorizes the program evaluated. We have 
found in some preliminary discussions with committee staffs that it is 
helpful to them to be able to select the information pertaining to par- 
ticular legislation. We have also found that while the titles of the 
evaluations give them some indication of the things that they might want 
to look deeper into, they need more information in order to make these 
selections. 

Chart on Descriptive Abstracts of an Evaluation 

Chart 3 illustrates for one of the evaluations in the Department 
of Labor list how the evaluation can be abstracted to show such things 
as the scope. Other information which can be included is the location 
where the study can be obtained, the period of time covered by the eval- 
uation, and possibly any major conclusion reached. 



71-867 O - 76 



280 



C5 



&Z 



dZ 


ca 


eu 


L."^J 


k!^l 


j~~ 


t£i 


«tC 




— 1 


to 


UJ 


S3 




Wf~ 


C^3 


czi 


N™* 










o? 


c 3 


UJ 


On 


EX. 


(SS 


« . u 


c J£ 




S33£ 


r *. : 


X 


' 3 


ktj 


H— 




e-:~^ 




S3 





to 






>— 








SI 


«3: 

=3 




—J 


Cr: 


f=l 


> 




F-* 


CD 




1_U 


OS 




Q. 

cc 




U-l 




■cc 




re 


«aC 


GO 








—J 


■»■* 








u. 


«£ 








es 


rs 










CS 






■>*■ 


1— 


cd 




r-» 


■— t»P 


cs 




CVj S 


CM 


UJ 


o_ 







*2 °= 2 

• . j ^ CO CO 

< o t: li.cc 

c3§^ °{Jj 

i§^ si 

s E g IB ^ 

==:*=< 5; tt! ^ 



era r^ 



=*=• CX3 -•■- 
C~> ^Z CO 

E-£ al L= 



C=> CO 







CO 




c:> 






«^ 


CO 


•a: 


CC5 




>■" 


■«*■ 


S3 




C j 


' n 


eg 


; ; 


jjsj 


c-j 


. .' 


( '3 


UJ 




cs 


c '■ - j 


O 


Cka 


UJ 


c ra 


r. J 


!-'J 




t!."-» 


L:~ 


5 


CO 


ca 




1— 


c 


, 


tJ 


ro 


13 




*3j 


ro 


Cvj 


«s£ 


— 


CO 


; /j 


CnJ 






CO 




c<; 






CO 


«sC 


OJ 






L'J 




ca 






zjc 


CO 





r— 
CO 



oo 



c :> 



co cJ <* 

CO «? =* =3 « 

>- n en _ 3 
J*J r- cm °? co 
o_ CO uj 

cc -J rb"" e-a co 

2E 
co 

CO 



"- ro «sj 



63 A-T 2= - 

I — -» CO ■ r 

cr> =»• c-o "^: cr> co : 

»— *— *■*■ ^j- ir> 

- -■- ^ u- w en ;,: S 

-co H -co«cc^-h_ c ir 



281 



cd 



CD 



ca ~j ^ 



CO 

rr r~D 

CD U.I 

E CO 

C DC 



re 

C_3 



a." 

CO 



Li S £2 Jr E; co 

F g g LS == t L J 

SO " «=> C_3 5- p- 

£2 CD c/> C3 gj 

— to « ■* E2 « 



CO 



:' C W W 

-'-; cj ti H — - 



CD C/3 



cs 



00 



CO 

f^ ** 

CD CO 

>: c3 
rtj co 



en *s 






o_ c_> 



c -: 

CD 






CO 



en 



c" "- a 00 h 

'-; I j O C O ° ^ 

*: c: o u-J =£ ** 

^~* 1— u^ -> , •--> ui 

<-° =3 . -ia H 

1— zz cnj ■* ** co 

rvi :=* • - o- « cs > 

- - 3H- z: cr> ^Z 

I— -V J= = *■ 2= tT . OC 
^ ^ < iH > < a *« 

= a ui == s « g g 

t; en £> . 



g < s 



U ;- ^ 



CO "J 



UI < W C3 - ^ 5 

Lo ° -J >* 

^r LU 
CO 



CO 

CD 



H W w 

OK 3> S£ 

e= -s !T: — : cd li- 



co 



CO 

CD 



CD ° 


>^ 


CO 

CO 


ct= — ' 


CD 


< 


-a: e 3 


2E 




a. cd 


LU 


LU 


1" 


CD . 


D» 


CD 


CL. 




Z 


cc 


LU 


~~* 


CD 




CO 
CO 


CO 


t 


LU 

CD 



CO 



ir> 



►= •" CD r- ^ e= 

^ es " -S S ^ =3 

S £ ~ £ £ S 5 



282 



We will consider also whether certain features of the methodology 
need to be included in this kind of information. For example, users 
of the evaluation may be particularly interested in the goals speci- 
fied for the program which normally are the basis for the choice of 
effectiveness measures. 

Chart on Criteria for Measures 

In our work on standards for evaluation, we have looked into pos- 
sible criteria for the choice of measures. Chart A shows a number of 
such criteria which are not original with us, but have been specified 
by a number of evaluation staffs working in the field. When an eval- 
uation is being planned, there are trade-offs which must be made among 
these criteria; e.g., the simplest, lowest in cost measures may not be 
the most sensitive measures or the ones which measure unique features 
of a program best. The evaluation staff will need to understand the 
users need to make these trade-offs efficiently. This list is an ex- 
ample also of the kind of standards and criteria which our Evaluation 
and Modeling Sources staff will be using in appraising the quality of 
selected evaluations and models which are of particular interest to 
committees. We have found in our preliminary discussions with commit- 
tee staffs that when they select from the inventory, evaluations of 
interest, that they will be interested in our help to determine whether 
these evaluations can be trusted. 

Evaluation and Modeling Sources System 

Most of the staff on these activities are systems analysis types 
and we have viewed this activity as a system. As Chart 5 shows, we 
begin with the evaluations or models, where they are in the agencies, 



283 






^w- 




UJ 
















i — 












;, _ 




*-Z 












£2 

1. j 




r -z 




UJ 

— 1 








«r£ 




O 




£^ 




, t 




^ 




CD 




CO 




>- 




"^^^ 




*s*i 


UJ 






CD 




CO 
CO 


UU 

r i 


UJ 




UJ 




UJ 


CO 


1 - j 




UJ 




CD 


CO 


-■■-* 


c •;. 




— 






*a: 


UJ 


UJ 

1 — 


1~j 
cp 




=3 

UJ 




UJ 


C3 


5£ 


— -3» 


U- 


UJ 

— 1 


CD 
CD 




c5 

CD 


CD 


CD 

CD 


CO 

CD 


cc 


UJ 

co. 




CD 


tz 


." ~ 


^""^ 


u 


c 3 




«— 


» s 




; .- 


PD 

._. J 


CD 
1 J 


ca 
i — 


UJ 

— 1 

CD 


CD 
CD 


UJ 
1— 


...... 


«c£ 


""*~- 


™— 


co 


'- — 






LaJ 


CD 
L-j 


UJ 
XJ 


c O 


;. * 


UJ 






-~j 


Q-. 


• " 


■ » 


■ 


CD 






UJ 


LO 


► - 


UJ 


Ji 


CD 






or 


ex: 


CO 


co 


ID 


•a: 










284 



EVALUATION INFORMATION SERVICE SYSTEM 



AGENCY 
PROGRAMS 



AGENCY 

EVALUATION 

PROCESS 

EIS MONITC'iiNG 
OF AGENCY 

EVALUATION 
PROCESS 



PROGRAM 
EVALUATION 
INFORMATION 



SELECT!;;;!, 

APrTiAisAL, a;;d 

asse$s:;e;:t. 

pnoon::,! evaluation 

INFORMATION 



ANALYTICAL USERS: 
1. GAO AUDIT SITES 

2. EXECUTIVE EVALUATION UNITS 

3. STATE EVALUATION UNITS 

4. OTHERS 





STRUCTURE BASED 
> ON 
CONGRESSION AL NEEDS / 



TAILORED 

QUICK 

RESPONSES 



CHART 5 



285 



and have established a two-way communication between our staff and all 
of these major staffs. We also have a continuous communication with 
our resident audit staffs at the agency sites. This establishes a ba- 
sis for us to monitor on a continuous basis any evaluation or model or 
any evaluation process if there is a need to do so. This is also the 
relationship through which we have conducted the recent joint Office 
of Management and Budget and GAO inquiry into agency evaluation activ- 
ities, which has provided a basic file of program evaluation informa- 
tion. 

As I- have already indicated, we are able from this basic file of 
information, to provide to any interested committee a list of the eval- 
uations which have been conducted of programs authorized under par- 
ticular legislation. When we go beyond that to assist the committee 
in appraising the quality of any evaluation or mode, we will also call 
on outside experts to supplement the knowledge or skills of our in- 
house staff. As we see it, this process will build an inventory of the 
most relevant and highest quality evaluation information which can be 
structured based on the needs of the various congressional users, to 
provide tailored quick response evaluation products for these users. 

In addition to the congressional users, GAO audit groups will use 
this information in reviews of program results. We expect that exec- 
utive agency evaluation staffs may wish to use this inventory to look 
for evaluations of programs which may be closely related to the ones 
they are evaluating. There are a number of State evaluation units in 
existence and we expect that they may find this inventory useful as 
well. We expect that will be a two-way street and eventually expect 
to have in our own inventory the best State level evaluations which 



286 



are available, particularly those which are evaluations of Federal 
grant programs. 

We believe the evaluation and modeling inventory I have described 
will have both direct use to the Congress, and will have indirect bene- 
fits in better analysis and evaluation for the Congress. 



287 



DISCUSSION 

The discussion following the GAO presentation centered on the expected 
utility of the joint GAO/OMB program information system. Mr. Marvin said 
that there was considerable and enthusiastic optimism regarding the genera- 
tion of program related information required for legislative evaluation 
and budgeting purposes. However, it will be quite a while before the 
two agencies develop a system which is both feasible and tailored to 
legislative information needs. 



288 



'PRACTICAL APPLICATION OF EVALUATION IN LEGISLATIVE PROCESSES", WILLIAM 
H. FOSKETT, CONSULTANT * 



. The following remarks on Program Evaluation: A Manual for Legislators 
and Legislative Staff will discuss: 

° what we mean by the term "evaluation", 
° the concerns and perspective behind the manual, 
° basic ingredients involved in using evaluation for over- 
sight, and 
° some cautions for using evaluation. 
My remarks represent one co-author's perspective which may or 
may not be shared by the other co-author (Harrison Fox) on any particular 
point. These remarks will not substitute for the manual itself, which 
goes into more detail on specific topics such as legislative language, 
leverage points, and the need for coordinating evaluation techniques with 
legislative processes. (The manual appears in the Appendix.) 

What "Evaluation" Means Here 



A variety of management research and social science research techni- 
ques come under the rubric of "evaluation". Though there are many difference 
among techniques, they are alike in that they are all empirical and systema- 
tic. Use of these techniques is not restricted to oversight of existing 
"programs". They may be applied before a program even exists (e.g., 
needs assessment) or they may cut across programs (e.g., policy analysis). 
Evaluation techniques may yield evidence on program implementation (e.g., 
process evaluation) , on the relative effectiveness of different approaches 
to a particular need or goal (e.g., planned variation): or, on the signi- 
ficance of program effects (e.g., impact evaluation). 

* Dr. Foskett assisted Dr. Harrison Fox, of Sen. Brock's staff in preparing 
the draft manual on program evaluation. Dr. Foskett is chief, Design and 
Assessment Division, Child Development Associate Consortium, Washington, D.C. 



289 



The research tools involved In evaluation are the same as those 
used in academic theory building. However, in evaluation these tools are 
driven by the framework of political issues and public policy decisions. 
Evaluation is a loose collection of empirical tools for systematically 
gathering facts, examining them and using them to think through public 
action. 

Concerns and Perspective Behind the Manual 

Evaluation offers great potential for strenghtening congressional 
oversight. But to realize this potential, legislators and legislative 
staff need to know how to use the tools of evaluation in service of legis- 
lative needs. An ability to apply evaluation in legislative processes need 
not require technical training in evaluation. But an aspiring legislative 
user of evaluation must know certain basic leverage points about the 
substance, focus and quality of evaluation research in order to integrate 
it into legislative strategies and tactics. The intention of this legislative 
users manual of evaluation is to acquaint congressional readers with the 
leverage points of evaluation in such a way that evaluation can be applied 
more powerfully and constructively in legislative work. 

Securing the Congressional Stake in Evaluation 

This evaluation user's manual is written on behalf of the congressional 
stake in evaluation. The style, timing and substance of evaluation has 
been influenced mainly by Executive branch management and policy planning 
needs or by academic notions of research. The evaluation profession, 
evaluation research institutions, and the concepts of evaluation have 
developed in service of executive clients, not legislative clients. Until 
legislative evaluation users' needs are strongly defined and established in 



290 



practice , staff should give close and thoughtful attention to evalua- 
tion research done for use in the legislative arena. The evaluation 
community does not already know how best to serve legislative clients. 
It will have to learn; users will have to teach it. 

The manual does not present or discuss the evaluation activities of 
the Executive Branch. This omission should not be interpreted as diminishing 
the importance or necessity of evaluation conducted on behalf of Executive 
planning and management needs. Nor should it be construed as minimizing 
the importance of Congressional and Executive cooperation and coordination 
in a wide range of evaluation activities. The evaluation activities of the 
Executive Branch should not be displaced or wholly duplicated in the course 
of obtaining Congressionally-oriented evaluation services. But as an 
evaluation user, Congress has some interests which it can better define 
for itself. This manual seeks to open the way for Congressional definition 
and pursuit of its own distinctive interests in evaluation. 

The interests and perspectives of the Congress and the Executive branch 
differ by design. The perspective of this manual assumes that a balance 
of power among co-equal branches of the government is desirable. But a 
variety of factors has created an imbalance among the legislative and 
executive branches. One factor, the information explosion, has flooded 
Congress with torrents of data. Another factor, the increasing complication 
of our Nation as it has grown in size and experiences, imposes many more 
demands on oversight nowadays. 

The Executive Branch has been relatively better staffed and organized 
to deal with the complexity and the explosion of information that 
characterizes our times. The result has been a marked loss of relative 
congressional power as events and Executive Branch development outstripped 
the evolution and practice of oversight. The cure for this imbalance is 



-291- 

not to diminish executive Branch capability to deal with the volume and 
complexity of information, but to increase congressional capability 
to deal with the volume and complexity of information it needs for over- 
sight. 

New Forces Come Into Play After a Law is Passed 

The interests, the perspectives and the pressures which coalesce in 
Congress to raise an issue or to pass a law differ from those which operate 
in the Executive Branch after the law is passed. Evaluation within 
Executive agencies is an adjunct to good management and should be en- 
couraged and supported despite the discomfort it sometimes brings to 
bureaucracy. But some agencies actively resist doing internal evaluation — 
and are even less likely to cooperate actively in developing evaluation 
evidence for congressional overseers. Even when an agency is actively, 
genuinely committed to internal evaluation, the priorities and plans 
which take shape within the Executive do not necessarily jive with those 
of Congress. Evidence which might indicate need for internal improvement, 
or need for actions other than those proposed or taken by the agency 
for other reasons is unlikely to be promptly and clearly provided 
voluntarily by even the most diligent agency evaluation units. Nonetheless 
practical factors (e.g., the burden of data collection) preclude complete 
duplication of evaluation efforts by the Executive and Congress. Better 
access to data for Congress, stronger planning for oversight, and stronger 
technical approaches (e.g., independent simultaneous statistical analyses 
of shared data) can do much to alleviate these problems however. 

Using Evaluation Constructively 

Evaluation should be used to discover what things work, and how 
successful efforts differ from similar but unsuccessful efforts. Data 



292 



produced by evaluation should be used to sort out those things which can 
be accomplished from among the many worthy goals and efforts which compete 

for scarce resources. Specific evaluation techniques suitable for a 
variety of legislative concerns are available. But none of these specific 
techniques can or should substitute for traditional congressional investi- 
gation. Evaluation is not a "scientific" alternative to that congressional 
or journalistic investigation which draws attention to flaws and shortcomings 
to mobilize political forces. 






Legislative Precedent for Evaluation 

Congress has provided the authority for Executive evaluation in a 
wide variety of legislation but relatively little of this has been tailored 
to oversight needs. Though most experience and legislative authority for 
evaluation has focused on Executive Branch management usage rather than 
congressional oversight usage, Congress does use evaluation on an occasional : 
ad hoc basis (e.g., the computer simulation of alternative funding formulae 
for ESEA Title I done by C.R.S.). Some of the techniques familiar to 
Congress (e.g., staff investigation teams, polls of constituents) are 
sufficiently similar to special evaluation techniques that in many cases 
these could be made much more powerful and productive than they already 
are. 

Key Elements of Evaluation for Legislative Users 

Evaluation is one oversight technique among a whole array of oversight 
techniques which Congress has at hand. Evaluation is not a substitute for 
these other techniques. Evaluation (in any of its specific types) should be 
coordinated with other oversight techniques in a whole oversight strategy. 



293 



Used tactically within an oversight strategy, properly selected and coordinated, 
evaluation techniques can produce a flow of concise information which strengthens 
other oversight techniques and is valuable in itself. 

This happy outcome will not result merely from innate characteristics 
of evaluation itself. Successful use of evaluation in legislative processes 
requires attention and control by evaluation users. User control, however, 
does not mean making technical judgments by political fiat, but rather 
properly distinguishing and protecting political judgements from technical 
information. User control and attention should keep the evaluation re- 
searcher targeted on the substance and time-frame of his client's needs. 
It must not be exercised to censor results. 

Evaluation Resources for Congressional Users 

Oversight evaluation can originate in at least three different settings 
which make a difference in access to evaluation resources: 

- in the history or language of a particular bill or act, 

- within the work agenda of a particular committee, and 

- in an individual legislator's personal office. 

The resources available vary under each of these different origins. 
In the history or language of an act, it may be possible for Congress to 
obtain evaluation resources through the evaluation staff of an Executive 
Branch agency. The various congressional agencies (i.e., CRS, CBO, OTA, 
GAO) and services (e.g., House Information System) can provide resources 
needed for evaluation. Availability varies with the agency involved and 
the source of the request for resources. 

When working in an individual legislator's office more resourcefulness 
is required to obtain evaluation services. But it can be done well and 
powerfully if the office is willing to assign a staff member to locate and 






294 



monitor the necessary professional evaluation resources in settings out- 
side the Congress and its agencies. Such resources can often be obtained 
at no cost or low cost if the office is in a position to use fellowship 
programs, university research centers or faculty or other concentrations 
of personnel trained in evaluation research. But to preserve his own 
control there must be good user liaison and monitoring of such resources. 
Through various computer index and access systems 9 e.g., SCORPIO) or the 
quick-response staff at Library of Congress, some evaluation reports can 
be gotten "off the shelf." But chances are that these reports will be 
lengthy, mostly irrelevant and out of date for your concerns. Such faults 
are not with the retrieval systems which are regularly updated, but are 
due more to lack of adequate user monitoring when the evaluation was 
being done. 

What is Takes to use Evaluation in Oversight 

Using evaluation on an ad hoc, catch-as-catch-can basis can have 

limited but dramatic effects occasionally. But this approach is more 

mischievous than effective in gaining substantial changes. If you wish 

to use evaluation systematically to give power and substance to oversight 

activities you will need: 

o Professional staff assigned to locate, monitor and focus evalua- 
tion resources and activities and coordinate them with oversight 
activities; 

o access- to professional evaluations , i.e., assistance of evaluation 
specialists working outside the congressional staff in a variety 
of institutional settings; and 

o data — either raw data collected specifically from the activities 
being overseen, or secondary data collected originally for other 
purposes but reanalyzed to suit oversight needs in the specific 
situation. 



295 



Coordinating Evaluation Techniques and Legislative Processes 

Assuming the capacity to put the above resources together, how and when 
can they be used? The kind of evidence needed depends on what the particular 
oversight question is. The particular evaluation technique to be invoked 
depends, thus on the kind of question being asked. If at least the most 
important, predictable oversight questions are framed well in advance it 
will be much easier to provide for and get the type evaluation evidence 
needed. 

When you choose to bring that evidence into play is a matter of poli- 
tical judgement — although the natural life of legislation would make cer- 
tain questions premature or irrelevant at certain points. 

A brief guide to evaluation techniques which provide evidence best 
suited to answer some general types of oversight questions is shown in 
Table 1 (on the next page). Beside each question are listed a variety of 
evaluation techniques which can generate the type of evidence needed to 
answer that question. This table can be used as a guide to the kinds of 
evaluation that should be called for in legislative language or history. 
Or, it can be used in an ad hoc approach to assist in finding the most rele- 
vant types of evaluation for the type evidence you need. 



296 



TABLF 1. 
EVALUATION STRATEGIES AND TECHNIQUES WHICH KAY HELP TO AKSWES OVERSIGHT 



IV lEt-ISLATIVE ACTIVITIES. 



OVFFSICHT questh 



EVALUATION STRATECIES ASH ' 



/ 






1. need, assess.eot: opinion surveys: Delphi ,„,„,«. 


2. What kind, at talog, »ul« bo <„. « re-*, the proM «, 


7. policy analysis: evaluation synthesis; ,„for-at,on retrieval . 


3. Mut evidence Is there to Indicate that each of these reredlcs could i 


,. inforaa.lon retrieval; pilot testing; evaluation «,„<„„,.. 


4 - earjfevjsir.ts.arrtfe ,. jl* *. ....J 


-^-•l" 1 8; """•""'/benefit analysis; 


s (services) directly? To fund slate or local aotwrma-cs 
to develop or supply remedies? To provide incentives for the private 1 


5. policy analysis: PrMriCA, ,:^ OT 


6. Which combination ol reaedy and Tederal role best suits the 
goals and Intentions of a Bill or Act? 


, p„ot testing; planned variation; policy analysis; rei „«„ ,_ 


7. Do the goals and Intentions of a Bill or Act catch up with the 

nature o( the ptoblca, the reaedles and the Federal role envisioned 
by the Bill or Act? 


7. policy analysis; POLITICAL J0BQ 


6. Does the Sill/Act say what Confess Intends cjcjrlj enough (In legl- 

to set criteria against which you can ccopare Congressional Intent 
with an Ae^ncy's pla-,s. pr^in l-pler.entatlon and budgets? 


8. POLITICAL JVDCW.EKT In consultation with Independent evaluation specialises 


S. Are Ajc^cy goals and plans cleat enough for you to tell 11 they 
are consistent with Coneresslcral goals and Intents! 


,. pre-.sse^nc of ev.luablHty; certificate of t.adlne.s for evaluation 


iO. Are Ajency prograa plans and objectives clear enough to evaluate Tor 
raecuej ve-Cranc. 1 ) policy review and prograa nanageaent? What 


10. pre-assessaent of ev.lu.b.Ut,; policy ,na,ys,.; , 


11. Are program services being delivered in the type, aacunt, quality 
and tlne-frane that Congress intended? 


analysis approaches); prograa auditing; tine and iwotfcn stu-JIes; felue- 


What kinds of projects are not succeeding? How aany? How are 


12. success seaich; bocus operandi evaluation; institutional self-study 


13. Would the results reported for a prograa have happened even without 
the prograa? 


13. "experimental" progran/evaluat Ion design; nodus operandi evaluation 


14. Is the program effective In meeting the goals or needs set foith 


U. effectiveness evaluation; adversary rval^Cm; g^^^a - 


)5. Is the prograa effective cnou??) to solve the prcbleo that the 


15. lapact evaluation; needs assessment; public opinion survey 


are they? 


H. slde-effects .u.vey; ..goal-free- e.a on; 


17. What can a prograa ecan In personal, huaan tei.sl The bad? The good? 


.7. h„™„ sources accounting; „,.-,...„..: . . ,.. „ ..,.,,« ....,„ 


18. Has the probleo changed that was originally addressed by the Act7 








:0. Should wc continue the prograa? As It Is? with najor chanres? 





■JO 7 



Providing Legislative Authority and Direction for Evaluation 

To plan ahead for oversight of executive activities, the necessary 

evaluation provisions should be written into the language or history of 

an act. The more conspicuously this is done, the more likely that the right 

type of evidence will emerge at the right time. And, strong evaluation 

provisions often have the desirable side-effect of mobilizing Executive 

activities more conscientiously than when Congressional follow-up is not 

expected. 

To adequately provide for evaluation such language should cover 

the following points: 

" purposes of the evaluation activities; 

° who will conduct the evaluation research (i.e., the specific or- 
ganizational unit of the Executive or congressional agency chosen); 

°the subject matter of the evaluation (e.g., "measure public per- 
ception of the responsiveness of government at the Federal, State 
and local levels. . ."); 

" when each kind of evidence from the evaluation will be needed (e.g., 
evidence of program implementation or service delivery to be pro- 
vided yearly) ; 

" specific funding to support evaluation activities (if clear arrangements 
are not made funds for evaluation are often usurped by more comfortable 
or dramatic activities); 

" reporting arrangements (to whom, when, what format); and 

" access to data (especially important to secure when an executive 
agency will be conducting the evaluation, if you wish the data 
analyzed from alternative points of view) . 

Some appropriate phrases covering various particulars under each of 
these points may be found in Program Evaluation: A Manual for Legislators 
and Legislative Staff . If all the above points are covered in language 
and history, a sound basis for oversight oriented evaluation will be pro- 
vided. 






298 



These points may also be used to review the adequacy of oversight 
provisions in existing legislation. If any are missing it may be diffi- 
cult to oversee the legislation in other than a catch-as-catch-can basis. 
Broad, permissive provisions for evaluation can work, but will require 
special attention if they are to actually yield timely, relevant evalua- 
tion evidence. Overly prescriptive provisions can be equally, if not 
more troublesome — e.g., stipulating specific measures or statistical 
techniques that later prove susceptible to administrative manipulation. 

Technical Leverage Points to Monitor or Review 

Unless you are trained and experienced in doing evaluation research, 

to become too involved in technical details of evaluation will consume 

an inordinate amount of staff time and may disrupt or delay the necessary 

research work. However, legislative staff can and should have a general, 

critical familiarity with the substance of an evaluation on the following 

technical leverage points. 

-the goals or needs addressed by the activities being evaluated as 
defined in the evaluation; 

-the extent to which evaluated activities are actually implemented 
or delivered as described; 

-the actual measures that will be fed into the evaluation; 

-the logical plan for collecting, comparing and drawing inferences 
from the data — i.e., the research design; and 

-the kinds of statistical analyses which will be used on the 
data — there is usually more than one "right" approach. 

Goals 

Evaluators not accustomed to working in legislative settings place 
great emphasis on the need to state goals clearly. But often the goals of 



299 



legislation are broadly stated to obtain passage of the legislation. 
This need not and should not prevent evaluation. An evaluation can be re- 
ferenced to the results of an empirical needs assessment, to detailed program 
plans submitted by an executive agency for approval of an oversight committee, 
or to legislative history that clearly stipulates the reference points 
intended. 

Service delivery 

Evaluation of program effectiveness or impact is unsuitable without 
prior evidence to establish that the program was actually delivered as 
described. Tactics such as "certification of readiness for evaluation" 
will help prevent the error - of inferring that an inadequately implemented 
program would not work if adequately implemented. 

Measures 

Measures should be pertinent to the kinds of issues Congress was 
concerned with in enacting the legislation, as- well as consistent with the 
goals, needs, or program descriptions to which the evaluation is referenced. 
Multiple measures should be used wherever feasible. 

Design 

The logical plan for collecting, comparing and interpreting the 
data should be able to rule out or minimize alternative explanations for 
the results obtained. "Experimentation" evaluation is the strongest 
technique for ruling out alternative explanations of results. 

Alternative explanations 

The possibility that findings yielded by an evaluation are an artifact 
of a particular statistical technique can be minimized by conducting al- 
ternative, simultaneous statistical analyses of the same data. Results 



300 



from such competing analyses of the same data should be compared for 
similarities and differences of findings they yield. 

Some Cautions For Using Evaluation 

The Data Burden . Request for data inundate participants, staff and 
managers of Federal programs at the state and local levels. Responding to 
such requests diverts resources required for program operations. Ordinarily 
the data compiled do not benefit the program, because the data are organized 
into categories that are used for some Federal-level administrative pupose. 
The expense of collecting and compiling the data is borne at the local level, 
ordinarily without any Federal support. 

Legislators and their staff contribute to such burdensome and wasteful 
data collection in two ways. First, two different purposes for data col- 
lection are often mixed: 

a. to identify and pursue individual cases non-compliance, 
malfeasance or project failure in order to stimulate legal action 
or exert pressure on behalf of individual interests, 

b. to oversee programs or jurisdictional areas (e.g., welfare 
benefits) on the whole for the purposes of effecting program- 
matic improvement, authorization, appropriations, or gauging the 
need for new legislation. 

Second, legislators and staff are prone to rely far too heavily on one-number 
indices of program implementation or effectiveness. 

The pursuit of data to redress abuses in individual cases should be 
gathered in traditional, casework, investigatory ways. It is extremely 
wasteful to tie up national data collection with cops and robbers tactics 
which do not provide adequate data for consideration of broad areas of 
problems and performance. Reliance on one-number indices usually goes hand- 
in-hand with this approach. The result is a proliferation of low-quality 
data of very limited use for broad legislative action. 



301 



Executive agencies must also share some of the blame for insufficient 
planning of data needs and usage. Failure of interagency coordination 
may cause local respondents to answer the same type question with 
different configurations of data for different surveys. 

Evaluation Failure or Program Failure? Past experiences (e.g., the 
evaluation of Headstart) have taught that flaws in evaluation research 
design should be eliminated before drawing conclusions about the success 
of failure of that which is evaluated. Failure to establish that a 
program was implemented as described or premature measurement of program 
effects are two of the simpler evaluation failures. Beyond these are 
the more esoteric, but important problmes associated with measurement, 
research design and statistical techniques. 

Evaluation bias . Evaluation techniques are empirical and usually 
objective, but no;' unbiased . Evaluation biases arise from necessary 
working assumptions, from technical judgements that must be called and 
from other legitimate sources. Barring the unlikely elimination of 
biases, it is wise to think of the findings of evaluations more as if 
they were legal evidence than scientific fact. Law has developed a 
variety of procedures and safeguards for dealing with evidence that can 
be most helpful in working with findings from evaluation. The applica- 
tion of select legal concepts to use of evaluation findings is being 
developed by Robert L. Wolf, currently at University of Indiana. 



302 



DISCUSSION 

Several points about priorities for doing and using evaluation were 
raised in the discussion following the Fox/Foskett presentations. It was 
noted that frequently political considerations and political expediency 
delimit both the selection of programs to be evaluated and the use of 
evaluation information in decisionmaking. The use of evaluation in de- 
cisionmaking might be enhanced if congressional committees and Federal 
agencies developed a system of priorities for evaluation. Such priority 
statements should outline specifically the questions which should be 
answered in the evaluation. Additionally the quality of evaluation 
research might be improved if simultaneous evaluations were conducted 
of the program. Similarly it was noted that there already are a wealth 
of evaluation reports and secondary analyses of many programs. Staff 
might find it useful to review these reports before embarking on an 
evaluation in order to improve the formulation of questions about the 
program and to enhance cost/effectiveness in doing evaluations. 



303 



"THE STATE OF THE ART OF PROGRAM EVALUATION", 
CLARK C. ABT, ABT ASSOCIATES 



Ladies and Gentlemen, my topic this afternoon is the state of the art 
of program evaluation. I'll try and cover as briefly as I can something 
about what program evaluation can do and can't do, particularly from the 
legislative aspect, how the program evaluation state of the art advanced 
in the last decade, who can do program evaluation, what we've learned 
about how it should be done, how quickly and cheaply it can be done, 
and some future directions. 

First of all in terms of what program evaluation can do and what it 
can't do, I make some assumptions of rationality that more information is 
better than less and that there is a high cost to uninformed decision 
making. These are assumptions; they are not always operating explicitly, 
but ideally would be. Cliff Graves mentioned some criteria at the Office 
of Management and Budget in terms of what evaluations should be able to 
do — they have to be timely, relevant, and accurate. We have empirical 
evidence that evaluations can be as timely as two weeks. Recently the 
Office of Technology Assessment operated some evaluations, and one on 
the ERDA took two weeks, another one on offshore exploration which had 
significant impacts on leasing policy took three weeks. So the ratio- 
nalization that we don't have time to do evaluations because it takes 
years and we have to make decisions in weeks is not valid. Evaluations 
can be done in weeks. 



304 



Years ago, I used to be involved in doing technical intelligence 
evaluations of large scale weapons systems and we usually had a week to 
respond to what the Air Force called "quick reaction capability require- 
ments". We found that in general these one-week evaluations of new 
intelligence information on systems developed about 90% of the informa- 
tion about program effectiveness that we got in a further year's addi- 
tional study. I'm not saying that's always the case in social research, 
but a great deal can be done in a very short period of time. 

In terms of meeting the criteria of relevance, the question is, 
"Can program evaluation answer questions of relative benefits and costs, 
and of cost-effectiveness and impact of programs?" I believe that eval- 
uations can, although they frequently do not, and I'll get into some 
of the obstacles to their answering the relevant questions in a few 
minutes. 

We can measure cost-effectiveness and cost /benefit of any kind of 
program. Let me draw the distinction between cost-effectiveness and 
cost/benefit. Cost effectivness is easier; some criterion of effective- 
ness is set, such as advance in reading scores in an educational pro- 
gram, and the amount of advance per unit of cost is the cost-effectiveness 
or, more correctly, the effectiveness divided by the cost. It's basi- 
cally a measure of the desired program output over cost input. 

Benefit/cost is harder to compute because here we are computing 
benefits and costs in commensurate, usually dollar terms. We're trying 
to estimate what amounts to computation of return on investment, so that 
we can compare different programs. It is hard to compare years of 
reading achievement per dollar with percentage reduction in cancer 



305 



fatalities per dollar — two completely different programs in education 
and health. If you have benefit/cost ratios for both, or return on 
fungible resources invested, you can compare those and when you can 
compare across programs, ideally, you can optimize the whole portfolio 
of government investments by what the economists call equalizing marginal 
utilities. 

All programs have a production function which usually shows grad- 
ually increasing productivity per investment up to some point of de- 
clining or diminishing returns to scale, at which point they taper off. 
Ideally one would invest in all the programs at the accelerating part 
of the curve, and when they start to flatten out, disinvest and invest 
in other programs that are still accelerating their productivity. One 
can't do that unless one has benefit/cost measures. We do know how to 
get those with program evaluation. 

We can also operate within programs to determine the components 
that are strongest and weakest and thereby improve programs. We can 
evaluate past programs through summative evaluations; we can evaluate 
present programs through formative evaluations in which continuous feed- 
back is provided the program to continue to improve it; and we can 
estimate the impacts and cost-effectiveness and benefit /cost ratios 
of future programs, either with simulation or if there are too many 
variables involved and our theory is inadequate for simulation, through 
pilot programs and planned variation experiments. 

We can determine the past, present, and future impacts of programs 
in various dimensions of concern to voters and policymakers — physical 
impact, social impact, economic impact, political impact, impacts on the 



306 



actual physical state of the environment, on the state of people's 
opinions, on their market response (we're measuring right now the 
elasticity of demand and supply for housing under conditions of housing 
allowances), the psychological impacts, the behavioral impacts. 

We can do all of this to resolve conflicts of program choice, 
to make decisions more rationally about whether to start up a program, 
whether to modify, expand, or contract a program, or whether to terminate 
a program, and we can aid that same set of decisions within a program 
about program components. 

What we cannot do with program evaluation is create new programs, 
make decisions, make the assumption of rationality come true, and get 
users or consumers of program evaluations to ask the right questions. 
We also cannot make value judgments that have any credibility where 
these are decisive program decisions. 

The most important limitation of program evaluation is that we 
can't get people to use it — at least not consistently — and there 
are three main reasons why we can't. Keith Marvin mentioned that in 
a survey of government agencies of why there wa6 little or no benefit/ 
cost analysis in budgeting, the answers were that it's hard to measure 
benefits and costs, there is not enough staff, and there is not enough 
time. I'll mention something about the staff and time resources later. 
Concerning it being hard to measure benefits and costs, I think there 
is plenty of economic and other theory that enables us to measure the 
benefits and costs of any program. The next major breakthrough in pro- 
gram evaluation will not be in methodology, I believe, because we have 






307 



sufficient methodology to do the job now, but rather in government 
utilization — particularly in the cross-cutting agencies of 0MB, GAO 
and Alice Rivlin's new organization. 

The three main obstacles to actual program evaluation use , I be- 
lieve, are goal conflicts , communications failures , and knowledge gaps . 
On goal conflicts, part of the problem is that there is not a clear 
agreement about the purposes of evaluation between the sponsors, the 
operators, and the users. For example, HEW may sponsor an evaluation 
that Congress will use and that some independent research organization 
will actually operate. They don't necessarily all have the same in- 
terests, and in fact there is a functional tendency to look at it dif- 
ferently. Evaluators want clarified goals and clarified consequences. 
Political actors who sponsor evaluation, and particularly who sponsor 
programs to be evaluated, sometimes need to reduce the distinctions 
and to muddy them in order to build winning coalitions that will get 
programs through. So they may not have the same interests, at least not 
at the same time, as evaluators. 

There are many communications failures among those three groups, 
partly because they don't always speak the same language. I won't be- 
labor that whole issue, but I think the main reason is because there 
are unclear statements of specifications for the operators of program 
evaluations. I have some specific suggestions which, if they were 
worked into legislation requiring program evaluation, might reduce the 
communication failures between program evaluators and program evaluation 
users. 



308 



Most frequently the work statements of program evaluation requests 
for proposals describe what they want done and how but not why , and there 
is very little information to the bidders about the actual purposes of 
the evaluation, partly for the very realistic reasons that purposes 
differ and that might be a source of conflict. With that overspeci- 
fication of the means of execution of program evaluations, we have 
support for non-competent bidders because they have their work laid out 
for them, and a great constraint and disincentive to innovation on the 
part of competent bidders. 

On the other hand, some work statements are so vague and some 
mandates for program evaluation are so global and unspecific that one 
gets involved in a guessing game of what is really intended and the most 
persuasive salesmen win the work rather than the people who might be 
the most competent. In terms of the overspecification error, the counter- 
measure to that, I believe, is limitation of the specifications to out- 
puts of program evaluation and constraints when obtaining those outputs, 
without definition of populations to be sampled, sample sizes, research 
design, and so forth, which are best left to the program evaluators to 
work out. 

With respect to the error of underspecification, the trouble here 
is that there is often a confounding of competence testing of potential 
program evaluators with contractual definition of the work output. It 
is desirable here to separate the two and make competence testing of the 
potential program evaluators a completely separate part from the actual 
specification of the work. That can be done by giving an exam or re- 
viewing records or asking for a specific competence demonstration, and 
then separating that from the actual work to be done. 






309 



Work, outputs should be defined in operational terms. Most of the 
enabling legislation for program evaluation that I have seen is extremely 
vague about the very specific questions it wants answered, and here I think 
the work of Bill Foskett and Harrison Fox 57 / is quite useful in helping to 
give you an idea of the kinds of questions that need to be specified. It's 
very hard to evaluate evaluations for whether they've been relevant and 
answered the questions if one doesn't know what the questions are. So le- 
gislation calling for program evaluation should have very explicit statements 
of what the minimum set of policy questions and program questions to be 
answered should be. 

Now I make a distinction between asking program evaluation questions 
and insisting that all programs have specified goals. Joe Wholey at the 
Urban Institute insists that you can't evaluate a program if you don't know 
what its goals are. I think he's certainly correct in that it's much easier 
and better evaluating programs whose goals are specified, than where goals 
have changed over time, or their goals are bogus goals that mask a loose 
coalition of a variety of goals that are actually operating. So if one re- 
quires a clear specification of goals before program evaluation can be effec- 
tive, one immediately opts out of most program evaluations. 

There is no requirement, however, that we can't have specific ob- 
jectives for program evaluation, even if we can't detect specific pro- 
gram goals. In other words, one can see a program, let's say a health 
maintenance program in a rural community, and it may never have bothered 
to put its goals together and one can conceive of various goals that are 
desirable and say, now the object of the evaluation of this program is to 
determine whether this rural health program made any difference to 



5J.' See the Appendix. 



310 



Work outputs should be defined In operational terms. Most of the 
enabling legislation for program evaluation that I have seen is extremely 
vague about the very specific questions it wants answered, and here I 
think the work of Bill Foskett and Harrison Fox is quite useful in help- 
ing to give you an idea of the kinds of questions that need to be 
specified. It's very hard to evaluate evaluations for whether they've 
been relevant and answered the questions if one doesn't know what the 
questions are. So legislation calling for program evaluation should 
have very explicit statements of what the minimum set of policy questions 
and program questions to be answered should be. 

Now I make a distinction between asking program evaluation questions 
and insisting that all programs have specified goals. Joe Wholey at the 
Urban Institute insists that you can't evaluate a program if you don't 
know what its goals are. I think he's certainly correct in that it's 
much easier and better evaluating programs whose goals are specified, 
than where goals have changed over time, or their goals are bogus goals 
that mask a loose coalition of a variety of goals that are actually 
operating. So if one requires a clear specification of goals before 
program evaluation can be effective, one immediately opts out of most 
program evaluations. 

There is no requirement, however, that we can't have specific ob- 
jectives for program evaluation, even if we can't detect specific pro- 
gram goals. In other words, one can see a program, let's say a health 
maintenance program in a rural community, and it may never have bothered 
to put its goals together and one can conceive of various goals that are 
desirable and say, now the object of the evaluation of this program is 
to determine whether this rural health, program made any difference to 



311 



the health of the rural area resident — whether that was the program 
goal or not, it should have been. 

It's very possible to write evaluation goals into enabling legis- 
lation without requiring program goals. The incomplete or absent 
explanation of why the research is needed deprives program evaluators 
of the political context and comprehension of what questions are to be 
addressed. In case you missed putting down what the evaluation objec- 
tives should be by informing the program evaluators why the research is 
needed, they can at least then derive the evaluation objectives. So 
its very important to put those down to add to general context. 

Most evaluation results do not feed into program or budget deci- 
sions . The connection is mainly random right now. I have it from 
some highly-placed authorities who I am sure would not wish to be quoted, 
that there is no significant connection now, no planned connection be- 
tween outputs of program evaluations and the major programmatic budgetary 
decisions about programs. It doesn't mean it never happens — I'm 
just saying it doesn't happen usually and it doesn't happen on a planned 
basis. I think Cliff Graves pointed out, quite accurately, that program 
evaluations are often used in making program decisions; but I think we 
could argue that they should always be used and even if they are used, 
the results may be a little stale and if we had planned for the output 
of program evaluation to feed into program decisions, the results would 
be better. 

I would suggest that in any enabling legislation for program evalu- 
ations these be designed as decision-driven evaluations — that is, the 



71-867 O - 76 - 21 



312 



scheduling of the program evaluation results should allow sufficient 
time afterwards for consideration for a decision that has to be made 
using those results. Many program evaluations have conflicting goals, 
adding to the communications problem. These conflicting goals are 
reflected in either global work statements and specifications or very 
fuzzy ones. A way of dealing with this legislatively is to resolve 
the goal conflicts internally before writing the specifications for 
the evaluation, rather than expressing conflicting aims in the speci- 
fication for the work. If these conflicting aims cannot be resolved — 
and they often can't be — it is better to accept the conflict and say 
we will simply sponsor two kinds of evaluations that have conflicting 
objectives and they will be mutually independent and we will resolve 
conflict after we get the results of the evaluations, rather than to 
anticipate that conclusion, mush the issue, and try to get a compromise 
evaluation going. 

The knowledge gap is the third major reason for program evaluation 
being underutilized, the first two being goal conflict and communication 
failures. The main problem in the knowledge gap is that it's a gap 
existing on both sides, both on the side of the program evaluation 
sponsors and users — not necessarily the same group — and the evalua- 
tion researchers or producers, and neither group completely or fully 
appreciates the position of the other. 

Program evaluators and researchers rarely appreciate the full 
political context and the pressures acting on government decision makers 
to make a decision, and the kinds of sensitivities they have to respond 
to. On the other hand, government decision makers, unless they come from 



313 



a background of social science or evaluation research, rarely appreciate 
the possibilities and limitations of program evaluation research. As 
a result, the government sponsors generally ask for less than they 
can get in terms of speed, relevance, and validity, and don't insist 
on as high standards as they could insist on. The evaluators, in turn, 
are not as responsive and communicative with respect to the political 
needs of the government sponsors as they could be if they had the infor- 
mation that they need to be that way. 

One of the problems is that evaluators often live in a fantasy 
world of ideal efficiency. Cost-effectiveness and benefit/cost ratios 
are very important to them, and after one has been working very hard 
to get these things for many years, one tends to assume that's what the 
world really cares about. Then it comes as a great shock after years 
of research when one actually talks to a Congressman or a legislative 
aide and starts talking efficiency, and the other person seems to be 
backing away all the time and gets a distant look and the eyes kind of 
glaze over, and you know you're not communicating. Then, if you have 
enough time and you're a little sensitive and you draw the other in- 
dividual out, you find out that the concerns are really about what 
Harold Lasswell very ably defined years ago as the work of politics — 
who gets what, where, when, and how, and not how efficiently and not 
how effectively. 58 / 



sa / Lasswell, Harold D. Politics: who gets what, when, how. New 
York, McGraw-Hill Book Co., 1936. 



314 



Efficiency and effectiveness are considerations, but probably not 
the most salient considerations. They are used as levers or sticks to 
boost or depress various popular or unpopular programs, but they are 
not the central issue they seem to be the methodologists. So there 
isn't really a good mutual understanding here of the needs of evaluation 
research, and I think a lot of that can be addressed by simply more 
communication between potential users of the evaluation research and 
the evaluation researchers themselves. 

Often there is the sponsor who intervenes, and the sponsor may not 
have the user's best interests at heart. For example, the user may be 
the Office of Management and Budget to decide program allocations, the 
sponsor may be a sub-department of an agency that's trying to prove 
that a particular program is terribly cost effective in order to con- 
tinue and expand it indefinitely. Obviously here the user and the spon- 
sor have very different objectives. The sponsor has a — public relations 
is too crude a term — a support objective. The user has a comparison 
objective — these are very different goals for evaluation research. If 
the evaluator is too responsive to sponsors, he may defeat the users' 
purpose. All three groups have to interact in the process. 

How has the program evaluation art advanced in the last decade ? 
First let me suggest that it's advanced enormously but there have been 
hardly any theoretical breakthroughs. We've had an engineering advance 
but not a scientific advance. Take the analogy of nuclear weapons or 
nuclear power. The theoretical work on that was pretty much completed 
in the 30 *s but it took another 20 years and it's still taking time to 



315 



fully realize the engineering applications of that theoretical work. 
In the same way, most of the statistical concepts, the experimental 
designs, the instrument design issues, the operational designs of pro- 
gram evaluations were done in the 1940 's or earlier — many of them in 
World War II operations research, in earlier public opinion and mar- 
keting research, in survey research in the 1930 's developed for com- 
mercial purposes. 

The scientific work was done a long time ago, but in the last ten 
years, partly through the diffusion of the cost-effectiveness analysts 
from the Defense Department to the social programs agencies of the 
government since the middle 1960's, these methods have diffused through 
the rest of the government, and, with the increasing investment in social 
programs, the techniques have begun to be applied. A lot of the mistakes 
and errors we've made in social program evaluations have resulted in 
the rediscovery of things that have been known scientifically for 20 
years but were simply not known to social programs evaluators. 

A good example is the regression artifacts that limited the validity 
of the Westinghouse/Ohio evaluation of Headstart, which as you may know 
could not find any significant achievement gains in children as a result 
of Headstart, and had enormous policy impact in probably limiting the 
funding in Headstart, resulting in other programs to maintain the effect 
that extinguished too rapidly such as the Follow-Through program, and 
experimental planned variation programs, rather than massive national 
programs based on theoretical assumptions that were unproven empirically. 
One of its main effects was the methodological one showing very dramat- 
ically the great limitations that would result from a flawed experi- 
mental design; namely the lack of randomized selection of experimental 



316 



and control groups partly responsible for the regression artifacts, 
which made it impossible to attribute the findings reliably to the 
program. 

We've had a great deal of technological development in program 
evaluation research in the last ten years, with the diffusion of sur- 
vey research methods, experimental design methods, instrumentation 
methods, diffusion of statistical sophistication, diffusion of cost/ 
benefit and cost-effectiveness analyses to new applications from their 
original hardware technology and military applications to health, 
education, welfare, housing, criminal justice and so on, and this is 
still going on. We've had an increase in the sophistication and sen- 
sitivity to the limitations. 

The evaluation art has advanced to the point today where we can 
evaluate anything, we can determine benefit/cost ratios, we can de- 
termine relative and absolute cost-effectiveness — what we can't do 
is always get it and communicate our results. 

Another major change in the last ten years has been the widespread 
diffusion of large-scale data processing capabilities. Ten or fifteen 
years ago a lot of the survey research we carry out now was practically 
too expensive. In our housing allowance demand experiment we did a 
base-line survey of 100,000 to finally select 4,000 households for the 
randomly selected control group. Even with 4,000 households because 
there are quite a few variables, some of the cells just barely achieve 
statistical significance in size. This was done for a few million 
dollars within a few months. Before large-scale data processing facil- 
ities were available, it would have taken many years and probably tens 



317 



of millions of dollars — it would have been at least an order of 
magnitude more expensive. So we have enormously greater data collec- 
tion and data processing and reduction power at our disposal than we've 
had before, and we are not using all of it that we have. 

Given these advances in the state of the art in the last decade 
and what program evaluation can do and what it can't do, I'd like to 
quickly mention some requirements for doing it effectively in tele- 
graphic form. I just want to give you an idea of the richness and 
complexity of this field , and that there is just so much to be known 
and used here beyond what has been used. 

The evaluation resources of competent people, money and time are 
very scattered, and one of the things we need to do is concentrate them. 
Instead of a little bit of evaluation everywhere, we need to do enough 
evaluation to be timely, relevant, and valid and accurate for the big 
payoff items. The highest cost programs do not get proportionately 
concentrated evaluation now; we need to allocate in proportion to impact . 
We can define impact in a variety of ways — but generally it's func- 
tion of how many people are affected, how much, for how long, how inten- 
sively. We generally know that, other things equal, programs that 
affect lots of people have more impact than programs that affect just 
a few people. Programs that affect people deeply, like whether it'll 
save their lives or not, have more impact than those that are somewhat 
more aesthetic or cosmetic, etc. 

Evaluations often compare programs within agencies, but rarely do 
they compare programs with similar goals in different departments . We 
have about a dozen income transfer programs, and for each of those we 



318 



have eligibility and certification processes which have never really 
been compared across different programs. Government income transfer 
programs on the average have about 100% overhead — if FAP had gone 
through, of the $4 billion cost, about $2 billion of it would have been 
just for administering the program. A large percentage of that is for 
outreach and certification. All these different income transfer pro- 
grams do certification differently. There's never a cross-cutting study 
to analyze which agencies do it best and most cost-effectively and what 
they can learn from each other. 

So we need cross-cutting evaluations of processes and functions 
that are common to a great many different agencies programs. I think 
this is a major new role for the cross-agency missions of the Office 
of Management and Budget, General Accounting Office, Alice Rivlin's 
Office 59 / and the Congress. I don't see how this can be done com- 
pletely within the departments themselves. 

Evaluations are rarely scheduled inputs to budget decisions, so we 
advocate decision-driven program evaluations by legislation . Evalua- 
tions are often not focused on controllable policy variables, so that 
the evaluation results don't always help in budget decision making. By 
a policy variable I mean a variable you can do something about, such as 
the particular characteristics of a* program, as opposed to a contextual 
variable, something you can't do anything about, such as the percentage 
of people under the age of 30 living in a town — you can't easily 
change that. A lot of evaluation research for academic reasons tends to 
focus on contextual variables which are not under the control of policy 



59 / The Congressional Budget Office, created pursuant to P.L. 93-344. 



319 



makers, instead of focusing on policy variables that are. These could 
be legislatively specified. 

Evaluation sponsors and researchers rarely communicate directly 
with the most important consumers of evaluation . In about 20 years of 
evaluation research I've never had the opportunity to discuss the pur- 
pose of an evaluation with the congressional leaders or legislative 
assistants who are mostly concerned with the results of that evaluation 
— never. 

Some evaluation research is mi6designed technically. If it does 
employ experimental design, there is often no randomized selection of 
experimental and control groups, and thus one is never certain that 
the findings are not the results of a confounding of some external 
event or factor with the program that is supposed to produce those 
results. 

There's a lack of any control groups in too many evaluation designs, 
so that there is no assurance of what we call internal validity, that 
is, the attribution of effects to the program that is being evaluated, 
or external validity, the generalizability of the findings. There's 
also often a failure to recognize and compensate for the statistical 
artifacts that I mentioned before, such as regression towards the mean. 

There's a frequent failure to select the readily measurable measures 
of effectiveness , and this can be remedied by legislation to some extent 
suggesting these. This is a delicate point, not specifying but sug- 
gesting some of the measures of effectiveness that are of interest. I 
say suggest rather than specify, because again here the researchers may 
come up with more useful measures and may find it unproductive to be 
constrained to specific measures. We talked about the too broad focus 



320 



which can be corrected by legislation specifying the questions to be 
answered, too narrow focus which can be remedied by the enabling 
legislation not specifying methods but rather results desired and 
questions to be answered. 

Too short duration . I realize I talked about some evaluations 
being possible within two weeks, and yet some evaluations, particularly 
the kind that depend on developmental changes in human beings, may 
require longitudinal evaluations lasting years, because people just 
don't change that fast. And cross-sectional evaluations, that is, 
the instant snapshot comparing many things at the same time, simply 
will not get at the change in people over time with much reliability 
and validity. It is possible to simulate a longitudinal study with 
overlapping cross sections in time, but it's always much more complex 
and expensive and less reliable. So if we're talking about the im- 
pact of daycare children or the impact of an educational program on 
life opportunity, we really ought to have the patience to wait and 
have sufficient duration to measure the more subtle effects. 

The misdesign of instruments : usually through compression in time, 
a lack of pilot testing, and corrective feedback. Any commercial sur- 
vey research outfit usually has time to pilot test and check its survey 
instruments — this is not always possible if you insist on doing 
evaluation research within the annual budget cycle. Data gathering 
misdesign is also a problem, including inappropriate allocation to sur- 
vey resources, sometimes resulting in oversampling of the least sig- 
nificant groups and under sampling of the most significant groups. I 
draw your attention here to the Watts-Conlisk design applied to the 



321 



New Jersey Income maintenance experiment. This was a good piece of 
research, but has some limitations in this respect because most of the 
sampling was done of the population that was of minimum policy interest, 
rather than the reverse, because this particular sampling allocation 
model economized in this particular direction. 

I think the only kind of regulation we can look for here is self 
regulation in the field itself. Qualitative competition among the 
evaluators themselves could really deter faulty work in the field. 
For a government policy on this, I would relate awards and fees to a 
consumer's report on the quality of previous work . That always raises 
administrative problems about who is to judge, etc . Obviously, it is 
not in the interest of a lot of sponsoring agencies to want this, so 
we cannot count on them. But I think there are some other ways besides 
an internal regulation. 

On the overall problem of the knowledge gap , I think it is very 
hard to ever really appreciate the capabilities and limitations of 
evaluation from the outside without having done it . It is a kind of 
way of looking at things. I think it is unlikely that the intended 
users in the legislature and in the executive branch will really ap- 
preciate it until they get involved in it. How can we tempt them to get 
involved in it? This is a paradox, because as has been pointed out 
by Professor Haveman 60 7 and by Senator Brock, they are not particularly 
eager to have their own activities and their own processes evaluated. 



fin / Professor Robert H. Haveman, University of Wisconsin. 



322 



I think the next best thing is to get them to participate in some of 
the planning of evaluations. That gives a sense of control, a sense 
of power over things, and basically more sharing by the executive and 
the administration with the legislative branch of the actual setting 
up of program evaluations. It is still a sequential process now rather 
than an integrated process. I don't know how to do this, but I think 
that some of this can be done informally by the practitioners themselves. 

Getting back to the legislative sponsor of the evaluation, which 
is often very much frowned upon by the executives in the administra- 
tion, involves some risk which has to be taken here of informal con- 
tacts asking "Is this really what you want to find out?" 

In summary, I'd just like to describe some particular weaknesses 
of evaluation that we've noticed and that I think if reduced or re- 
moved could improve the usefulness of evaluation and applicability of 
the work to legislation. 

Some of these typical weaknesses, in no particular order, are: 

Lack of randomization in the selection of experimental and 
control groups, biasing the findings. 

Lack of any control groups, rendering the validity and 
generalizability of the findings completely uncertain. 

Failure to recognize and compensate for statistical arti- 
facts such as regression towards the mean. 

Failure to select the readily measurable measures of effec- 
tiveness relevant to the policy issues. 

Too broad a focus, and failure to limit the evaluation to 
program aspects that can be researched. in enough depth to achieve 
validity within the time and effort resources available — the 
"fising expedition" that generates a low signal-to-noise ratio. 



323 



Too narrow focus, with premature concentration on a few 
variables assumed to be significant at the cost of exclusion 
of potentially more significant ones. 

Too short duration to allow sufficient iteration of the data 
gathering efforts to narrow down to the critical issues in depth. 

Misdesign of questionnaires, often as a result of a lack of 
pilot testing that may be a result of a lack of sufficient time. 

Data gathering survey misdesign, particularly the inappro- 
priate allocation of survey resources that result in an over- 
sampling of the most significant groups, as was the case with 
the attempted application of the Watts-Conlisk sampling model 
to the New Jersey income maintenance experiment. 

Ignorance of or failure to explore relevant literature and 
data concerning similar programs or phenomena being evaluated — 
a point often stressed by J.S. Tukey. 

Failure to recognize and design around the Hawthorne effect 
and other types of self-fulfilling and self-selecting and de- 
selecting phenomena. 

Voluntarism in the selection of surveyed experimental and 
control groups leading to biased results, as when unsuccessful 
programs choose not to be part of an evaluation. 

Premature and inappropriate data collection efforts of 
perishable information that cannot subsequently be readily cor- 
rected, as in the collection of significant amounts of unanalyzable 
data on planned variation in the Follow- Through Experiment. 

Confusion of a program inventory for a program evaluation 
— the substitution of merely descriptive case studies for analysis 
of program effectiveness and efficiency with respect to goals and 
resources use. 

Confusion of needs assessment for an evaluation, assuming 
that determination of the relevance of program design to an 
established set of needs constitutes an adequate evaluation. 

Confusion of program outputs for impacts, as is frequently 
the case in school evaluations that boast of increases in average 
daily attendance that may or may not have any relationship to 
desired educational impacts such as increased life time earnings. 

Contamination of program results by formative evaluation 
procedures that directly influence program activities as part of 
the evaluation processes in non-reproducible ways. 



324 



Mis-selection of program goals and subsequent criteria of 
effectiveness, as in cases where the program operator's goals 
differ from those of the sponsors and possibly those identified 
by the evaluators. 

Some of the specific technical and managerial approaches which 

may help to correct these typical weaknesses of contemporary evaluation 

research are described below. 

Lack of Randomization 

The excuses most frequently given for lack of random assignment 
to experimental and control groups are evaluation timing, political 
obstacles and resource constraints. (There is still the all-too-common 
unexcused reason that "matching" will serve the same purpose, where it 
most often will not. This needs to be corrected by the further sta- 
tistical education of evaluation sponsors, designers and users.) 

The post-hoc timing of many evaluations render random assignment 
subject to the accessibility of a "natural" control group which satisfies 
this requirement. Since most post-hoc evaluations are concerned with 
treatments specifically addressed to a particular "target" population's 
problem, it is unlikely that such a "natural" control group that sat- 
isfies other experimental conditions can be readily identified after 
the treatment is already well under way. One policy for avoiding this 
problem is to require evaluations to be designed and executed concur- 
rently with the programs to be evaluated. 

Political obstacles to random assignment exist when a treatment is 
perceived as very desirable, and untreated controls perceive themselves 
as being discriminated against. Where "double-blind" experimental 



325 



approaches are not feasible (as in most social, as contrasted to bio- 
medical, experiments, because of treatment detectability by laymen), 
some forms of side payments to compensate controls for providing 
information while not receiving treatment have proven effective. In 
operating HUD's Experimental Housing Allowance Program's Demand Ex- 
periment, for example, Abt Associates Inc. has not experienced any 
significant political opposition from the randomly selected controls 
not receiving the allowance, probably because they receive a modest 
payment (much less than the housing allowances) simply to compensate 
them for supplying control information. 

Resource constraints are claimed to exist where random selection 
of controls is inconsistent with a high probability of obtaining sta- 
tistically significant quantities of both experimentals and controls 
for all the variables of findings for a smaller set of variables versus 
le86 validity (or unknown validity) for a larger set. In most cases, 
sponsors and producers of evaluation should require that the available 
surveying resources be distributed to achieve valid findings for what- 
ever set of variables can be accommodated, rather than invalid findings 
for a more ambitious set. Alternatively, exploratory or inventory- 
taking evaluations may require inclusion of all known variables but not 
generalizability of findings. In these latter cases, random assignment 
to controls, and sometimes even controls themselves, may be sacrificed 
for other research objectives, but the limitations of the resulting 
findings should be clearly understood and accepted by sponsors and 
evaluators at the outset. 



326 



Lack of Control Groups 

The absence of a control group is often justified by lack of 
data gathering resources, political infeasibility, and the irrelevance 
of comparisons. 

The first two obstacles have been discussed above. The issue 
of irrelevance of comparisons requires clarification. When a pro- 
gram's relative effectiveness and efficiency are to be evaluated, 
controlled comparisons are being developed, the experimental method 
need not apply. 

Sponsors of evaluation and research must distinguish among basic 
research, development, test, and evaluation programs. Basic research 
may or may not require control groups. Development and testing rarely 
do, and evaluation usually does. Where the evaluation aspires to 
validity, generalizability, and replicability, control groups should 
be insisted on. 

Failure to Recognize Statistical Artifacts 

Most of these kinds of errors of evaluation can be avoided by 
advanced training in statistics and research design. A straightforward 
policy for assuring such competence is to require it of the technical 
director of all major evaluation efforts. If this is impractical, a 
somewhat more cumbersome but probably effective approach would be to 
require all evaluation research designs to be reviewed and accepted 
as technically sound by a group of statisticians of proven competence, 
institutionalized within some government agency. OMB clearance of 
questionnaires provides both a good and a correctable model for such 
an approach. 



327 



Failure to Select Readily Measurable Measures of Effectiveness Rele - 
vant to Policy Issues 

This is often a problem where evaluators are more concerned with 
contributing to a discipline or field of knowledge then they are to the 
formulation of a policy or operation of a program. 

The most obvious corrective approach is to conduct a decision 
analysis of how the policies or programs of interest will actually 
be decided on, before the evaluation is designed. This decision anal- 
ysis should identify the relevant decision criteria and supportive in- 
formation requirements, so that the evaluation research can be specified 
to meet those requirements. 

Failure to select readily measurable measures of effectiveness is 
often the result of incomplete or insufficiently specific conceptualiza- 
tion of an evaluation research problem. For example, in an evaluation 
of the effectiveness of child day-care centers in meeting children's 
developmental needs, an incomplete conceptualization would include 
only children's activities in the centers, and neglect the parent- 
teacher-child interactions. Alternatively, an insufficiently specific 
conceptualization would call for measuring "all developmental activities' 
whatever that might mean. Priorities of data collection must be set to 
make sense with respect to a comprehensive conceptual model of the 
phenomena being evaluated, and then expressed in operationally definable 
terms. 

Too Broad a Focus 

This is generally a problem of either over-ambitiousness or under- 
appreciation of the costs of collecting and analyzing information 



328 



responding to specific questions, or both. It can be corrected, like 
most misallocations of effort, by setting clear priorities and esti- 
mating opportunity costs for specific tasks, maximizing the accomplish- 
ment of highest priority high productivity tasks. Sponsors of evalua- 
tion sometimes contribute to the native intellectual imperialism of 
some evaluators by urging the evaluators to find out "a lot about a 
lot of things", with the likely result that little that is new or 
useful is found out about even a few things. This process is en- 
couraged by competitive procurement, wherein bidders attempt to outdo 
each other in the technical accomplishments they promise. Within all 
too rough limits of credulity there are apparent rewards for promising 
too broad an effort at some presumably technically competent depth, 
and few disincentives to promising too much. 

Organizationally, much could be done to reduce these incentives 
to over-promises of excessive breadth by : (1) sponsors specifying 
the required scope or breadth and faulting bidders for going far be- 
yond it; (2) sponsors evaluating bids on the basis of the percentage 
of promised achievements actually realized by the bidder in completed 
researches; and (3) continuous and candid communication between spon- 
sors and researchers throughout the period of contract performance con- 
cerning research priorities and the cost and tradeoffs involved in 
realizing them. 

Note that a pro-forma monthly progress report does not often 
achieve the third objective. Face-to-face meetings between sponsor and 
researcher are needed to develop enough mutual trust and confidence for 
the evaluator to be able to risk admitting that the goals of the 



329 



evaluation had shifted and with the task priorities, without the re- 
searchers resorting to legalism or unresponsiveness. 

Too Narrow a Focus 

Partly in response to having been "burned" by the non-results of 
an evaluation too broadly focused, or because they think they know what 
the key variables are, sponsors and evaluators may insist on too narrow 
a focus and prematurely exclude some data and variables that might have 
yielded significantly different results. Truncated samples, arbitary 
and unexplained elimination of outliers, and ignoring of non-responding 
data sources are all examples. 

Ideally, given sufficient time and resources, all evaluation re- 
search should be executed in at least three cycles — the first broad 
and shallow to maximize the probability of capturing the critical 
variables, the second more narrow and with an in-depth analysis of 
the hypotheses generated in the first cycle, and then a number of 
almo6t-as-deep parallel analyses to check external validity and reli- 
ability. 

Since this kind of iterative process is not always affordable, 
it may be helpful to attempt to compress the cycles to within a given 
evaluation, or to use analogical other efforts to simulate the first 
and third cycles. In any case, a thorough and broad literature search 
should reduce (but will not eliminate) the probability that a too nar- 
row focus has excluded potentially significant factors. 

Too Short Duration 

This is sometimes the sponsor's fault in insisting on a one-year 
duration for contracting or budgeting convenience, and sometimes the 






330 



researchers' fault when through poor planning much of the contract time 
is wasted and the long lead time tasks must be compressed to a duration 
inimical to their successful completion. 

What is needed here is some standardization of the lead times of 
the lengthier evaluation research tasks, such as field testing of ques- 
tionnaires, data reduction and cleaning, etc. Further, there needs to 
be a better appreciation of both the advantages and the elapsed time 
requirements of longitudinal research — classical "before and after" 
treatment evaluations in school settings are impractical within one 
year, since it is unlikely that this duration permits the necessary 
planning and design before baseline data gathering or the necessary 
data reduction and analysis time after the after-treatment data is 
collected. 

Misdesign of Questionnaires 

This or other data- gathering instruments can be corrected by pilot 
testing, provided there is sufficient time available. Much of the pilot 
testing time might be saved if standards of competence in instrument 
design were more strongly insisted upon by sponsors, since almost every- 
one believes that they can design an effective questionnaire. 

There is remarkably little questionnaire research built into most 
evaluations, yet the results of the entire effort are usually criti- 
cally dependent on the quality of these instruments. This is an ac- 
tivity in which there appears to be very little technique transfer 
between the masterful practitioners and the neophytes — again, because 
neophytes believe it is easy (and it ±s^ easy to do it badly) . Standard- 
ization of competent instrument design approaches and their dissemination 



331 



might help. Sponsors could raise standards by requiring evaluation 
researchers to submit examples of previous instrument designs, to- 
gether with critiques of their strengths and weaknesses in the light 
of actual use. 

Data Gathering Survey Misdesign 

Here again, many marginal evaluators fail to be aware of or exploit 
the great mass of survey research and sampling methodology is quite 
adequate for meeting the technical demands of most evaluation researchers. 
The major problem is that the full competence of this sub-field of social 
science is rarely fully applied. 

The state of the art under-utilization problem can be addressed by 
sponsors imposing higher standards of survey competence on the proposing 
evaluation researchers. Appropriate course work and experience in sta- 
tistics, sampling, design, and instrumentation should be specified for 
survey task leaders. 

For the more sophisticated survey design problems, the uncer- 
tainty concerning the optimal choices of defining factors may be par- 
tially overcome by pilot testing and/or simulation. 

Ignorance of the Relevant Literature 

This is correctable by effort, insistence on it and demonstration 
of its results. 

Failure to Recognize and Design Around the Hawthorne Effect 

Again, this is a matter of education of the producers and consumers 
of evaluation research, and the enforcement of competency standards 






332 



reflecting the best that is known concerning research design. Campbell 
and Stanley 61 / offer numerous methods, such as the Solomon four-group 
design, for dealing with this similar testing artifacts. 

Voluntarism in Selection 

This is often as much a policy as it is a technical issue. Spon- 
sors wishing to avoid offending program operators may back off from 
insisting on evaluation of programs or sites that resist it. Sponsors 
may intentionally or inadvertently select sites, cases, or projects in 
biased ways influencing the results of evaluation. 



61/ Campbell, D.T. and Stanley, J.C. Experimental and quasi- 
experimental designs for research on teaching. In N.L. Gage, ed. 
Handbook of research on teaching. Chicago, Rand McNally, 1963. 
(Reprinted as Experimental and quasi-experimental design of re- 
search. Chicago, Illinois, Rand McNally, 1966.) 



333 



"EVALUATING EDUCATION PROGRAMS ARE WE GETTING ANYWHERE? 

JOHN W. EVANS, OFFICE OF EDUCATION * 



From the Educational Researcher , published by the American 
Educational Research Association. Remarks delivered were 
based, in part, on this paper. 



334 



Evaluating Education Programs 
Are We Getting Anywhere? ' 



JOHN W. EVANS 
U.S. Office of Education 



Now seems to be a time when 
basic reassessments are in 
order; so it is only appropriate for 
those of us concerned with educa- 
tional evaluation to take stock of our 
own endeavors and ask whether we 
are really getting anywhere in our 
efforts to assess the effectiveness of 
educational programs, or whether 
all the current talk and frenetic ac- 
tivity is much ado about nothing. 
The answer to this question is by no 
means obvious, even though it is the 
kind of rhetorical question that pa- 
pers like this typically answer with 
rosy if vague and overqualified bro- 
mides. I'm going to hedge and 
qualify too, but my basic conclusion 
is that we are getting somewhere. 
I believe important progress has 
been made in recent years in edu- 
cational evaluation in a variety of 
ways which I intend to specify, but 
the educational evaluation scene is 
not an untroubled one. Far from it: 
there are serious new problems that 
threaten the progress that has been 
made, and I intend to talk about 
those also. 



A Brief Look at History 

We can begin with some simple 
and chastening assertions. First, the 
history of significant evaluations of 
educational programs is brief and 
thin. In this respect educational 
evaluation is of a piece with the 
evaluation of social action programs 
generally. If we look back over the 
history of Federal efforts in the so- 
cial program area, even back into 
the period of the New Deal and on 
up through the Great Society pro- 
grams of the sixties, we are forced 
to acknowledge that virtually all the 



original decisions by the Congress 
and the Executive Branch of the 
Federal government to initiate pro- 
grams in the areas of education, 
manpower and poverty, and the 
later decisions to continue, expand, 
or terminate these programs, were 
taken with scarcely any knowledge 
of the size, character, and location 
of the problems, or the likely effec- 
tiveness of proposed programs to 
remedy them. Once instituted, such 
programs were only rarely subjected 
to rigorous objective evaluation. 

Second, failure to evaluate edu- 
cation programs is a shortcoming 
not limited to the Federal Govern- 
ment. States and localities, which 
supply 95% of the funds for public 
education, have done little to evalu- 
ate the effectiveness of their school 
systems and educational approaches. 

Third, with a few notable excep- 
tions, academic social scientists, 
traditionally accustomed to the re- 
search style of the individual schol- 
arly grant, have been largely pre- 
occupied with «Jisciplinary issues 
and basic research, rather than 
"with actual evaluations of ongoing 
educational programs. 

Such have been our efforts to for- 
mally evaluate most of our national 
education and other domestic pro- 
grams. Yet, in less than ten years 
we have gone from a dearth of eval- 
uation activity to a situation where 
evaluation is now all the rage. Even 
though the number of professional 
association meetings currently being 
devpted to evaluation are exag- 
gerated indicators of the amount of 
useful, policy relevant evaluation 
now going on, there is no doubt that 
the change has been real and sub- 
stantial. 



What has accounted for this rela- 
tively sudden upsurge, and does it 
amount to real progress— progress 
in conducting sound evaluations and 
making them pajt of the policy 
process? There is no simple answer 
to this question, but from my vant- 
age point there seem to be several 
important factors which have ac- 
counted for the increased concern. 

First, there is the long-run, cumu- 
lative effect of the presence and 
force of social science in our society. 
Social scientists have long been 
chiding administrators, policy 
makers, and Congressmen to rely 
less on subjective and political rea- 
sons for making decisions, allo- 
cating resources, and developing 
programs, and instead to make 
more use of the research methods 
and findings of social science. These 
entreaties have often turned.out to 
be more rhetorical than substantive 
once the challenge was taken up, 
But they have not gone without ef- 
fect on the policymakers, who have 
been made to feel increasingly 
guilty about not formulating policy 
and making decisions in a more 
rational way. 

Second, there has been a gradual 
transformation in the intellectual 
make-up of the kind of people who 
have found themselves in both ap- 
pointed and elected offices. While it 
remains true that raw, unreasoned, 
political interest still is the main 
factor in many decisions made by 
both the Executive and Legislative 
branches of the Federal Govern- 
ment, it is also true that the last 10 
to 20 years have seen a significant 
increase in the number of people in 
such positions who want to attack a 



September 1974 



335 



problem by asking about the scope 
of its dimensions and the effective- 
• ailable methods of treat- 
ing it — that is. people who want to 
try to rationalize the policy process. 

Certainly another important fac- 
tor in accounting for the upsurge of 
concern with evaluation at the Fed- 
eral le\el was the implementation of 
the Program Planning and Budget- 
ing System (PPBS). This approach 
to analyzing and making decisions 
about program budget le\els is a 
radical departure from the tradi- 
tional incremental approach. It 
turns the focus away from the stand- 
ard administrative budget categories 
toward the objectives, methods, and 
outcomes of programs. Such a shift 
automatically brought the need for 
data on program effectiveness to the 
fore. 

Finally, there is the accountability 
movement in education. It is hard to 
know whether this movement is a 
cause, an effect, or merely an indi- 
cator of the increased interest in 
evaluation In any case, once pres- 
ent, it has become a force in its own 
right 

But what have all these changes 
in analytical approach and ex- 
pressed concern amounted to be- 
yond heating up the atmosphere and 
expanding the rhetoric? What have 
they resulted in that allows one to 
conclude that some actual progress 
in educational evaluation is being 
made? To answer these questions 
let us look at three aspects of the 
evaluation process: first, the in-puts 
and resources: second, the meth- 
odology: and third, actual evalua- 
tion studies and their results. 

Some Indicators of Progress 
Resources for Evaluation 
and the Avenues of Impact 

There have been major increases 
in the wherewithal required for eval- 
uations to get done, and important 
improvements in the organizational 
location of the evaluation function 
in the Federal Government's deci- 
sion-making apparatus. As the larg- 
er social changes I noted earlier 
have heightened concern with eval- 
uation generally, the Congress has 
tired of listening to requests for in- 
creased appropriations based on 
anecdotes and testimonials, and has 
increasingly demanded that Execu- 



tive branch agencies produce data 
on the effectiveness of their pro- 
grams. For its part, the Congress 
has substantially increased funds 
and personnel to the domestic agen- 
cies for evaluation. In 1965. the 
Departments of Labor and HEW 
had available less than S5.0 million 
for program evaluation. By I9"4 this 
figure had increased more than ten- 
fold to more than $50.0 million. 

There have been equally impor- 
tant changes in the organizational^ 
location of the evaluation function. 
As those of us who have worked in 
program agencies can testify, one of 
the indispensable prerequisites if 
evaluation is to impact on decisions 
is that it must be an integral part of 
top management's decision-making 
structure. Yet, evaluation, like re- 
search, has often been buried in the 
bowels of program agencies and not 
only has received the leftovers in 
fiscal and personnel resources but 
also has had little opportunity to 
make a meaningful input into the 
policy process. Even this is now 
changing. Most of the major Fed- 
eral agencies now have an Assistant 
Secretary for Planning and Evalua- 
tion or its equivalent, and the shoe 
is now on the other foot. Instead of 
having to plead for more money and 
people and the chance to participate 
in the decision-making process. e\ al- 
uation specialists are now under 
pressure to produce and to justify 
their claims of utility and relevance 
As mans of us have found out, de- 
manding our place in the sun is a lot 
easier than justifying it. 

Perhaps the most important thing 
to come out of all these resource 
increases and organizational changes 
"is that the basic dialogue of man- 
agement has begun to change from 
considerations of how big a pro- 
gram's budget should be and of the 
constituency pressures for its con- 
tinuation, to considerations of ob- 
jective evidence of performance and 
indicators of program effectiveness. 

The Lse of More Sophisticated 
Methodology in Evaluation 

1 want to touch briefly on two 
important methodological advances. 
The first is the appearance of efforts 
in large scale national evaluations 
to use the classic model of experi- 
mental design with randomized 



treatment and control groups. 

Since virtually all education pro- 
grams have their committed advo- 
cates and strong detractors, and are 
in this sense inherently controver- 
sial, it is inevitable that evaluations 
of them will also be controversial. 
Any evaluation that finds a pro- 
gram successful "will be attacked 
by the program's detractors, and 
any evaluation that finds the pro- 
gram unsuccessful will be denounced 
b> its advo'cates. As Rossi (1972) 
■has put it, "No good evaluation 
goes unpunished." In such cases, 
the controversy will not be waged 
directly over what is truly at stake, 
namely the disagreeableness of the 
findings, but instead will take the 
form of an attack on the validity of 
the evaluation's methodology Acri- 
monious debates will rage through 
the daily press and the professional 
journals over sample size in- 
adequacies, non-representativeness, 
culture-biased measurement instru- 
ments, failures to meet the assump- 
tions of parametric statistical mod- 
els, and the like. Since no evaluation 
can be flawless, especially those car- 
ried out in the real world of class- 
rooms and communities, evaluation 
specialists will never escape these 
post-evaluation debates — nor. in- 
deed, should they. But to increase 
the justification for using formal 
empirical evaluations as a basis for 
policy decisions, it is important that 
each evaluation be as methodologi- 
cally strong as possible. 

The major weakness in most eval- 
uation designs relates to the use of 
control groups. The feature of edu- 
cation evaluations that has proven 
most vulnerable to both well-moti- 
vated and not-so-well-motivated 
attack is the comparability between 
treatment and control groups. Once 
evaluations move beyond the primi- 
tive efforts to conduct site visits or 
simply collect data on a before- 
after basis, the evaluator and the 
design he employs must confront the 
fundamental problem of providing 
some estimate of what would have 
happened in the absence of the pro- 
gram he is evaluating. There are. of 
course, a variety of ways to deal 
with this problem, which include 
comparison with national norms, 
comparison with previous years' 
scores, the use of a matched com- 



336 



parison group, etc. But the effects of 
education programs are seldom dra- 
matic, and the small differences 
they are likely to make can easily be 
either overestimated or missed en- 
tirely by comparing the treatment 
group with a non-comparable group. 
The only truly satisfactory way of 
dealing with this problem, of course, 
is through randomly assigned treat- 
ment and control groups. Campbell 
(1969) has written eloquently and 
extensively on this issue, urging that 
educational and other kinds of pro- 
grams be structured at the outset to 
allow this kind of evaluation design. 
He and I have debated the question 
of how acceptable evaluations are 
which fall short of this standard 
(Campbell & Erlebacher; Evans & 
Schiller, 1970). My view is that 
there will be many instances where 
this obviously stronger design is not 
feasible, and that rather than throw 
up our hands and withdraw from 
the arena because we cannot have 
random assignment, we must carry 
out whatever kind of evaluation is 
feasible and useful within the time 
constraints of the policy process and 
make the best use of it we can. It is 
my experience that even fairly prim- 
itive designs are likely to provide 
better data for decision-making 
than the subjective impressions and 
partisan arguments normally used. 
My point here is simply that while 
we must be willing to do what we 
can to improve the decision-making 
process, I nevertheless agree with 
Campbell that we should certainly 
strive to use the superior method of 
randomized experimental design 
whenever we can. And I would re- 
port that the Evaluation Unit at the 
Office of Education has been able to 
mount two such randomized design 
evaluations of the Emergency School 
Assistance Program in the last two 
years. One is completed and the 
other is still in process. In a forth- 
coming paper in which "he analyzes 
efforts to introduce and assess in- 
novations across a broad variety of 
fields from medicine to education, 
Mosteller notes that this is appar- 
ently the first time in education in 
which a major evaluation was car- 
ried out using this type of design. 

The other methodological ad- 
vance I want to comment on is the 
emergence of experimentation as a 



precursor to the full fledged intro- 
duction of major new programs. 
This could be one of the most im- 
portant developments in our time if 
it takes hold and is actually used — 
and those are two big ifs. 

To understand what social or edu- 
cational experimentation entails and 
how it may be of great value in 
developing educational policies and 
programs, it may be helpful to look 
at what has happened in the field .of 
compensatory education. In the ear^ . 
ly sixties the country belatedly rec- 
ognized the existence of the dis- 
advantaged child. It was acknowl- 
edged that about a fifth of our chil- 
dren arrive at the first grade with 
educational deficits that are meas- 
urable even at that time; that as 
they progress through school the 
achievement gap between them and 
their middle class peers widens; that 
a great many do not learn to read, 
write, or calculate adequately; that 
they drop out of school in large 
numbers; and that as a consequence 
of these deficits in basic skills and 
credentials they are unable to pur- 
sue postsecondary education or to 
form either a lasting or satisfying 
attachment to the labor market. 
What is worse, these educational 
disadvantages are passed on cul- 
turally to their children, and a sig- 
nificant portion of the society's pop- 
ulation is caught up in a cycle of 
intergenerational poverty. 

Once these problems were recog- 
nized, the sense of political urgency 
was irresistable, and the country 
rushed to pass major legislation and 
initiate, among other things, major 
early childhood compensatory edu- 
cation programs. In this atmo- 
sphere of rushing to solve the prob- 
lem, only the faintest of voices were 
heard asking the unaskable ques- 
tions: Did we really know what we 
were doing? Did we really have ef- 
fective program models and tech- 
niques which could remediate the 
educational deficits of these chil- 
dren? 

The evaluations that have since 
been carried out on these national 
programs, principally Head Start 
and Title I (e.g., Westinghouse, 
1969; and Wargo, 1972), indicate 
that we did not know what we were 
doing. While our ideology was laud- 
able and our motivations pure, our 



programmatic know-how was 
skimpy. 

The problem is that once large 
national programs are put into 
place, the political force of their 
authorship and the pressure from 
their constituencies for continued 
funding make it almost impossible 
to even acknowledge publicly that 
they may not be effective, much less 
attempt to alter them in some fun- 
damental, way to make them so. 
This is where a strategy of experi- 
mentation can be of great value. 
After there is broad recognition of 
a major educational or social prob- 
lem, if we can force ourselves to 
recognize that we may not know 
how to solve it, and if instead of 
going directly to a massive national 
program, we wiy initiate a con- 
trolled experiment in which we de- 
velop and test the relative effective- 
ness of alternative programmatic 
techniques, we can reap a number of 
benefits. First, if the results of that 
experiment show that we have not 
yet achieved effective program 
models and techniques, it is politi- 
cally possible to admit this and go 
back to the drawing board to devel- 
op them. Moreover, we can take 
considerable satisfaction in the 
knowledge that we are not com- 
mitted to the continued expenditure 
of large resources on efforts we 
know to be ineffective. On the 
other hand, if the experiment is suc- 
cessful, we can go forward with a 
large national service program rea- 
sonably confident that the massive 
resources we will be devoting to the 
problem will have a good chance of 
actually making a dent in it. 

The logic of using experimenta- 
tion as a precursor to national 
program implementation may seem 
compelling enough, but the rush to 
adopt this strategy has not exactly 
been a stampede. Analyzing this 
situation, Timpane (1970) has made 
a series of observations on the po- 
tential of social experimentation 
which I have jokingly called "Tim- 
pane's law." His conclusion is that 
if there is enough interest in some 
problem to support a major social 
experiment, then the interest will be 
so great that no one will be willing 
to wait for the conclusion of the ex- 
periment before passing legislation 



September 1974 



337 



to implement a national program. 
On the other hand, if there is not 
broad concern over the problem, 
then there won't be enough interest 
in Congress to support the funding 
of an experiment on it. Either way 
there is no experiment. 

There is truth as well as humor in 
Timpane's law, but some significant 
efforts at experimentation in educa- 
tion are nevertheless occurring, and 
we should not overlook their impor- 
tance both as early prototypes of 
what could become a fundamentally 
new way of approaching the devel- 
opment and initiation of education 
programs, and for the contribution 
they have made to the particular 
educational issues they address. I 
would cite two such efforts. The first 
is the Follow Through program and 
the second is the OEO experiment 
on Performance Contracting. 

Follow Through was originally in- 
tended to be a follow-up service 
program to reinforce whatever gains 
were made in Head Start; but by 
the time its first appropriation in 
1968 had passed through the vari- 
ous budget cutting phases, the initial 
request of $120 million had been 
reduced to $15 million. Realizing 
that it made no sense to mount a 
service program that would address 
only one percent of the target popu- 
lation, the program staff shifted the 
focus away from service delivery to 
the development and evaluation of 
alternative compensatory education 
models. This effort at educational 
experimentation has been plagued 
by staff shortages, administrative 
difficulties, and continued unclarity 
over what Follow Through's true 
mission is and how it should be car- . 
ried out. Nonetheless, despite these 
problems, some evaluation findings 
are now beginning to emerge that 
are precisely the kinds of outcomes 
we would expect from a planned 
variation experiment. Some of the 
program models are showing the 
ability to produce cognitive and af- 
fective gains that are larger than 
those we have seen in most com- 
pensatory education programs. Oth- 
er models are producing gains that 
are just about what one would ex- 
pect from the normal school expe- 
rience, while still others are appar- 
ently so ineffective that the children 
in the control group are educational- 



ly belter off than those in the model 
programs. If these findings hold up 
in the subsequent waves of the long- 
itudinal evaluation, we should have 
a much better basis on which to pro- 
ceed programmatically in the area 
of early childhood compensatory 
education. 

The OE experiment on Perform- 
ance Contracting grew out of the 
kind of situation which should call 
for an experiment". About four v years 
ago a number of educational tech- 
nology firms were promoting the ■ 
ability of their techniques to pro- 
duce large gains in reading and 
math among disadvantaged chil- 
dren. Interest in performance con- 
tracts began to sweep through pub- 
lic school systems with large popu- 
lations of educationally disadvan- 
taged children. The attraction to 
performance contracting w;is based 
on a number of factors. It was at 
this time that the disillusionment 
about public education that flowed 
from the Coleman and later analy- 
ses was approaching its peak. The 
siren of performance contracting 
was especially seductive at this 
time because it said: "Not only is it 
possible to remediate the deficits of 
disadvantaged children, but we have 
the techniques to do it, we are ready 
to come into your schools and im- 
plement it, it is no more expensive 
than your present per pupil expendi- 
ture, and we will sign a binding con- 
tract with you which says that if we 
don't produce significant, independ- 
ently measured gains in reading and 
math, you don't have to pay us." 
Small wonder that these blandish- 
ments triggered a rush to the per- 
formance contractor's door. 

But there were also strident crit- 
ics of performance contracting, 
mainly the teachers' unions, who 
argued that performance contract- 
ing was an illusory panacea and 
that it would dehumanize the learn- 
ing process. Depending on who won 
the argument — that is, who shouted 
the loudest— it seemed that per- 
formance contracting was destined 
to be either prematurely buried or 
unjustifiably expanded into a na- 
tional movement. 

Noting that these unfounded 
claims and counter charges were 
precisely the circumstances that cull 
for an experiment, the evaluation 



staff at OEO designed and carried 
out such an effort, underwriting and 
independently evaluating seven dif- 
ferent performance contracting 
firms. The results of the evaluation 
showed that none of the perform- 
ance contract models was able to 
produce reading and math gains sig- 
nificantly better than the results 
achieved through the regular public 
school methods. It is hard to predict 
what trie, outcome of the debate 
would have been had the experi- 
ment not been done. 

Studies and Results 

Given that educational evaluation 
has shown progress in the funding 
support it has attracted, in the sheer 
amount of evaluation activity that 
is going on, in^ the organizational 
position that evaluators hold in gov- 
ernment agencies, in the demand for 
evaluation results by the Congress, 
and in the use of more sophisticated 
designs in the Conduct of evalua- 
tions, what has actually been done 
by way of major evaluation studies 
that have important policy implica- 
tions? It is the completion of such 
actual studies, after all, that is the 
outcome measure for evaluating 
evaluation. 

First, there is the Coleman Re- 
port itself which, while it does not 
evaluate a specific education pro- 
gram, nevertheless is fundamentally 
an evaluative analysis assessing the 
effects of what had traditionally 
been regarded as some of : the most 
important independent variables in 
the educational process. Notwith- 
standing the continuing debates 
over its methodological shortcom- 
ings, few would deny that it is a 
landmark study which has caused 
educational theorists to reassess 
their fundamental beliefs and strat- 
egies, and legislators to reexamine 
their unquestioned faith in educa- 
tional programs and appropriations. 

Second, the OEO evaluation of 
Head Start, usually referred to as 
the Westinghouse Report, which 
was also the subject of intensive 
methodological scrutiny and debate, 
is one of a number of studies of ear- 
ly childhood compensatory educa- 
tion programs which shook us out 
of our complacent belief that popu- 
lar, well motivated programs are 
necessarily effective in remediating 



338 



the educational deficits of poor 
kids. 

Third, an evaluation of the Emer- 
gency School Assistance Program 
found that this moderately funded 
and locally generated collection of 
projects was able to significantly 
increase the achievement levels of 
black male teenagers, and thus, by 
indicating that compensatory educa- 
tion in the public schools is possible. 
was a welcome contradiction to the 
largely negative findings of so many 
of the earlier studies. 

Fourth, and in the same vein, an 
early evaluation of the Upward 
Bound program found that this pro- 
gram was effective in persuading 
low income high school youngsters 
to attend college, in keeping them 
there, and in graduating them at a 
rate which made the program cost- 
beneficial. 

I have already mentioned the 
major evaluations of the Follow 
Through Program and Performance 
Contracting. I don't wish to extend 
this list indefinitely, mainly because 
I couldn't if I wanted to. But. I do 
want to make the point thai if we 
ask whether all the hoopla of edu- 
cational evaluation has amounted to 
anything more than increases in 
funds, data gathering, and profes- 
sional meetings, the answer is "yes " 
There is far less on the production 
ledger of educational evaluation 
than there should be. but indica- 
tions of important progress are by 
no means lacking. 
A Look at the Future: 
Prospects and Problems 

This recitation of progress makes 
things sound much better than they 
are. To be sure, the progress is real; 
but in the last few years a number 
of new problems have arisen that 
must be solved if the use of evalua- 
tion in the policy process is to prog- 
ress beyond its present promising 
but inchoate state. I call them new 
problems because it is important to 
distinguish them from the kind of 
problems we would have listed ten 
years ago. The review I have made 
so far should make it clear that edu- 
cational evaluators can no longer 
complain that they do not have 
enough money or people, or that 
they are not taken seriously by p" 1 
icy makers and legislators. More- 
over, I do not agree with those who 



argue that methodological inade- 
quacies of one sort or another pre- 
sent a major obstacle to the full 
flowering of educational evaluation 
as a policy instrument. It is not un- 
common for social scientists to dis- 
play handw ringing despair over 
their primitive methods and insen- 
siii\e measuring instruments, and to 
plead that an Einsteinian break- 
through in the social sciences is 
needed to put things right. My- own 
view is that we have a long way fo 
go in making full use of the tech- 
niques we now have before we are 
in a position to complain about in- 
adequate methods. 

The newer problems education 
evaluators face are of a different 
order: 

1. As educational research and 
e\aluation have proliferated, the 
people and institutions who are the 
objects of these studies have come 
under an increasing data collection 
burden — and are increasingly ex- 
pressing their resistance to it. It is 
no longer possible to assemble a 
battery of interview schedules and 
questionnaires and invade the 
schools. Extensive prior clearance 
and review are now required almost 
everywhere, and outright refusal to 
participate in studies is not uncom- 
mon The research and evaluation 
community is going to have to work 
out some collective way of dealing 
with this problem, for it is a real 
one. By the time school systems 
total up all the data collection re- 
quirements from Federal. 5 tate. 
local, and private sources, the bur- 
den often is a crushing one. 

2. Evaluation studies that involve 
collecting data on adu:: are en- 
countering increasing resistance at 
the interviewee level, particularly 
among minorities and the poor 
where it is now not uncommon for 
respondents to insist that they be 
paid for their time. 

3. The increased sensitivity to 
evaluation studies — both what they 
seek to find out and the amount of 
data they propose to collect — is re- 
sulting in a strangling growth of 
revifews, clearances, and advisory 
bodie«. The problem these mul"" 1 .' 
involvements and clearances pose 
«■-,- ln e evaluator are so great il • < 
threaten to prevent many evalua- 
tions from being carried out at all. 



4. As protests over evaluations 
arise, ostensibly over the type and 
amount of data to be collected, there 
is likely to be an increased politici- 
zation of these protests and their use 
as weapons in broader disputes be- 
tween local and Federal levels of 
government. 

5. As evaluation activity and pol- 
icymakers' interest in it have grown, 
there has also been an increased 
awareness at the program level that 
it is necessary to start taking evalu- 
ations seriously. This has had the 
unfortunate effect on some program 
officers and school administrators 
of increasing their unwillingness to 
participate in evaluation studies for 
fear of what will happen to their 
programs if the evaluation produces 
negative findings. 

6. Evaluations are increasingly 
encountering unrealistic expectat- 
ions on the part of policymakers 
and legislators with respect to both 
the speed with which evaluations 
should be mounted and completed, 
and the simplicity of the answers 
which are desired. A demand for 
evaluation has been created, and it 
is an increasingly insistent one. Pol- 
icymakers are beginning to display 
an irritated impatience with the 
elaborate trappings of careful de- 
- : -;n, longitudinal studies, and com- 
plex multivariate findings. J u "\ 
want to know whether or not a pro- 
gram is any good and they want to 
know it yesterday. As unrealistic as 
these expectations are. evaluators 
tnemselves probably must bear 
some of the blame for them. In their 

rly zeal to have the virtues of 
evaluation recognized and used, 
evaluators were almost certainly 
guilty of overpromising. 

This problem has already passed 
the stage of irritated impatience. 
Last year, the Congress cut the Of- 
fice of Education's evaluation budget 
in half and made large reductions in 
its statistics budget and in NIE's 
research funds. 

7. We are certain to see a lot 
more public debate of the kind 1 
noted earlier over the validity of 
evaluation methods and results. \n 
increasingly important and .ime 
consuming task for evaluators will 
be defending the evaluu'ions they 
carry out and their suitability as a 
basis for policy decisions. An unfor- 



339 



tunale by-product of such debates is 
the impression created among both 
policy-makers and the public that 
the mere fact such a debate is oc- 
curring means the evaluation must 
ipso Jaclo be faulty and therefore 
should be put aside. It is ironic that 
after a large scale formal evaluation 
has been put aside because of tech- 
nical questions raised about its 
methodology, policymakers and pro- 
gram officials then return to the old 
and familiar methods of making the 
decision or formulating the policy- 
methods that are highly partisan 
and subjective in nature. 

The seriousness of these problems 
should not be underestimated mere- 
ly because they are technical and 
procedural in character. Perhaps 
we can take some solace, however, 
in the realization that these are the 
problems of impact and success 
rather than the problems of neglect 
and disregard. 

The fact that such problems are 
being encountered is an indication 
of how far evaluation has come in 
the last decade. Educational eval- 



uation has gone from not being 
taken seriously to being expected to 
produce. It has gone from a condi- 
tion of no funds, people, or influence 
to one of being held accountable for 
producing valid and useful studies. 
It has gone from not having enough 
money to do evaluations to the 
technical and political problems of 
carrying them out. Some evaluators 
who have struggled so hard, to 
bring about these changes are noV 
wistfully wondering whether they 
wouldn't just as soon have their old 
problems back. As Oscar Wilde ob- 
served, there are two particularly 
dissatisfy ing things in life: the first 
is not getting what you want; the 
second is getting it. 

In sum, while I do not agree with 
the cynical view which holds that 
educational evaluation is largely a 
waste of time because its methods 
are too weak, because it will be for- 
ever unsupported, or because im- 
portant policies and decisions will 
be made in spile of evaluation find- 
ings, and while I believe that im- 
portant progress has been made in 



educational evaluation during the 
past decade, I nevertheless believe 
that educational evaluation now 
faces a new array of problems that 
are possibly more serious than the 
basic ones of getting the necessary 
resources to do. evaluations. These 
new problems are a strange mix- 
ture of logistics and politics, and 
they are in large part an outgorwth 
of the incteasing pluralism of Amer- 
ican society. If they are not dealt 
•with, evaluation will not succeed in 
making more than an occasional or 
marginal impact on educational 
policies and programs. If they can 
be solved, the general trend, which 
has only recently been established, 
can be continued; and the wider use 
of evaluation can make a major con- 
tribution to the setting of national 
educational policies, to the develop- 
ment of education programs, and to 
the allocation of scarce educational 
resources. 



Invitational address presented at the 
Annual Meeting of the American Educa- 
tional Research Association, Chicago, April 
18, 1974. These remarks focus mainly on 
Federal evaluation programs, but they also 
apply to state and local evaluation efforts. 



Rossi, P.. Testing for success and failure in 
social action. In P. Rossi and W. Williams, 
Evaluating social programs. New York: 
Seminar Press, 1972, p. 32. 

Campbell, D. T., Reforms as experiments. 
American Psychologist. 1969. 24,409-429 

Campbell. D. T & Erlebacher. A . How 
regression artifacts in quasi-experimental 
evaluations can mistakenly make compensa- 
tory education look harmful In J. Hellmuth. 
Ed. Compensatory education: A national 
debate. Vol 3. Disadvantaged child. New 
York: Brunner Mazel. 1970. 

Evans, J W. & Schiller. J., How preoccu- 
pation with possible regression artifacts can 
lead to a faulty strategy for the evaluation of 
social action programs: A reply to Campbell 
and Erlebacher, in J. Hellmuth. Ed., Com- 
pensatory education: A national debate. Vol 
3. Disadvantaged child. New York: Brunner 
Mazel, 1970. 

Mostellcr. F., Social Experimentation 
(forthcoming). 

Westinghouse Learning Corporation. The 
impact of head start: An evaluation of the 
effects of head start on children's cognitive 
and affective development. Vols. I and II. 
June 12. 1969. 

Wargo. M , et al. ESEA Title I A reanaly- 
Sb and synthesis of evaluation data from 
■1970. American Institutes of Research, 
ralo Alto, California. March 1972. 

Timpane. M., Educational experimenta- 
tion in national social policy. Harvard Edu- 
cational Review. 1970, 40, 547-566. 



340 



DISCUSS1CH 

Several methodological and organizational issues were raised 
following Dr. Evans' presentation. Like other speakers, he said 
that effective use of evaluation information requires that the re- 
search be sponsored by an office located strategically to influence 
the agency's planning and budgeting processes. It is important for 
methodological expertise to be built into the agency. However agen- 
cies should have staff capable of overseeing the methodological rigor 
of the study, and not leave this responsibility to extramural re- 
searchers. 

Questions were raised about the Office of Education's use of 
sole source contracting for evaluation research. Dr. Evans said 
that this practice has been discontinued and now the agency uses 
only competitive bidding to procure evaluation research. 

Like other speakers, Dr. Evans noted that evaluation is often 
hampered by the absence of precise legislative goal statements. This 
situation is being improved since agency staff have begun to estab- 
lish better working relationships with committee staff to get a bet- 
ter idea of intended program goals . 



341 



"EVALUATING EVALUATION: A CONGRESSIONAL PERSPECTIVE," ALLEN 
SCHICK, CONGRESSIONAL RESEARCH SERVICE 



As the concluding speaker at this CRS-sponsored seminar on pro- 
gram evaluation for Congress, I have the enviable task of evaluating 
evaluation. During the sessions of this seminar, we have been told 
why evaluation is a good thing and what Congress is doing to get more 
of it. We have been told that the executive branch is on its own 
evaluation trip and that there are new and sophisticated techniques 
for assessing the costs and effectiveness of Government programs. 
Everything seems to be in GO condition for a great leap forward for 
the evaluation business. 

But a bit of history should caution us that things might not be 
so easy. Although the Legislative Reorganization Act of 1946 canon- 
ized "continuous watchfulness" as the duty of all congressional com- 
mittees, legislative oversight still is not practiced in a regular or 
comprehensive fashion. Thirty years after the 1946 act, the promise 
of legislative oversight is far from realization and one cannot be 
sure that the impetus of the 1974 Congressional Budget Act and the 
House Committee Amendments will suffice to overcome the lack of per- 
sistent, indigeneous incentives for evaluation. 

There must be a reason why evaluation is so wanted and so unat- 
tained. My own hunch is that not having had to do with the wanting, 
that program evaluation is coveted because there is so little of it. 
The prize has become more valued as the stakes have escalated and as 
disappointments with program results have increased. Thus, within 
the space of 30 years, Federal expenditures will have climbed tenfol 



342 



from approximately $35 billion after World War II to the $350 billion 
plus projected for the bicentennial fiscal year. 

The Executive Is a Not So Evaluating Branch 

Self -evaluating organizations are administrative freaks ar.i t'r.ey 
rarely rank among the more successful or durable executive agencies. 
From time to time, organizations invest in stocktaking and retrospec- 
tion, but these tend to be mere pauses in the striving for security 
and growth. The exceptions shed some light on the normal impulse of 
self-promotion. During the sad decade of its short life, the Office 
of Economic Opportunity (OEO) excelled in its sponsorship and utiliza- 
tion of program evaluations, even to the extent of terminating some of 
its most promising initiatives when they proved to be ineffective. In 
OEO, evaluation came to be a substitute for action, a manner of exis- 
tence appropriate for a mendicant rather than an affluent organization. 
Lacking sufficient resources to underwrite regular operating programs, 
OEO applied its meager resources to small-scale experiments and pilot 
tests in a futile effort to stimulate social innovation. However, 
well-funded agencies tend to move into action before they have evalu- 
ative confirmation of the worthwhileness of their program innovations, 
and they continue what they are doing even if the results are unfav- 
orable. 

There are a number of reasons why agencies do not regularly eval- 
uate their own performance. 

(1) The Budget Problem . The annual budget process would seem to 
offer a hospitable opportunity for assessing results and applying them 
to next year's money decisions. On paper at least, each year an agency 



343 



has to be born anew and its course can be redirected by budgetary 
choices. Yet the straight fact is that budgeting is ill-suited for 
an evaluation role and that in many environments evaluation might 
have a better chance if it is separated from the routines of bud- 
geting. 

Agencies are not neutrals in the recurring budgetary wars; they 
are budget maximizers. The language of budgeting is geared to ad- 
vocacy and justification, not to the objective search for truth. Rath- 
er than face a life or death situation every year, agencies confront 
the budget with expectations of incremental — or better — growth and 
with confidence that their established level of activities and fund- 
ing is secure. Because they almost always receive at least as much 
money as they were given for the previous year, agencies have a weak 
incentive to review what they have been and to seek more useful ac- 
tivities. In the context of the annual budget struggle, a truly 
objective evaluation can injure an agency's budget position by in- 
viting outsiders (such as Congress) to reduce appropriations below 
previously fixed levels. If an agency deploys its evaluations to 
substitute new for existing programs, it runs the risk of losing its 
existing programs without getting the new ones. A case in point is 
the current effort by the Secretary of Defense to utilize savings 
from reductions in headquarters overhead for additional Army divi- 
sions. If Congress decides to convert the savings into fewer mili- 
tary dollars rather than more firepower, it will be difficult for 
the Defense Department to sustain this evaluation effort. 



71-8R7 ; - 



344 



The pace of budgeting discourages the application of evaluative 
findings. Budgeting is among the most routine of administrative pro- 
cesses, responding with cyclical regularity to timetables and dead- 
lines. It is not easy to cram evaluation into this fixed schedule 
of events. 

(2) Evaluation is an Unpleasant Task . Evaluation is not the most 
effective way to make friends and influence people within your own or- 
ganization. An evaluator quickly becomes an outsider within his own 
organization, one whose views and loyalty are suspect. There is good 
reason why evaluators are guaranteed in "inspector general" units and 
why the chief auditor of the United States is graced with a 15-year 
term. With normal participation in organization activities, the pros- 
pect is that an evaluator will be quickly seduced by the ethic and 
striving of the organization. It is not so much a matter of a person 
tolerating inefficient or unproductive work; rather, most people tend 
to embrace the identities and objectives set by their organizations. 

(3) An Organization is a Captive of its Clients . Once a program 
is established and benefiting its clients, an agency cannot easily re- 
direct its efforts. With their particular view of a program's bene- 
fits, the clients may have a different assessment than one emerging 
from more objective criteria of costs and benefits. There is an un- 
derstandable tendency for beneficiaries to believe that what's good 
for them is good for the U.S.A. Even comparatively new programs 
quickly fall prey to capture by their clients, for once a program 

is underway, the benefits are "vested" in their recipients and hard 
to terminate. This is one of the reasons why persons interested in 



345 



program evaluation tend to concentrate on pilot tests and experiments 
prior to full operationalization of the program. 

(4) Evaluation is a Difficult Task . Although there have been sig- 
nificant recent advances in evaluation methodology — for example, in the 
construction of samples and control groups — the routinization of eval- 
uation remains a distant possibility. Part of the problem is that eval- 
uative data are difficult and costly to secure; they are not ordinarily 
generated as a matter of organizational routine. Unlike conventional 
management information which relates to the internal operations of the 
agency, evaluative data frequently deal with what is happening outside 
the organization, and, hence, it takes a special effort — such as a sur- 
vey or experimental activities — to produce the wanted data. For example, 
a well-run health organization is likely to possess a great deal of work- 
load data such as patient-staff ratios, per patient costs, and number of 
meals served, but it is not as likely to be informed about the morbidity 
and mortality rates of the population it serves. 

Even when data have been secured, their interpretation poses nu- 
merous difficulties. Rarely are the findings so unambiguous as to per- 
mit only one interpretation. More commonly the data are open to various 
interpretations, depending on the assumptions that are used and the ob- 
jectives against which the results are measured. As the famous Head 
Start evaluations demonstrated, even a costly evaluation is vulnerable 
to methodological attack by persons who refuse to accept the official 
interpretations. A decade later, scholars and practitioners alike still 
are warring over the soundness and policy implications of the Coleman 
report. 



346 



(5) Fear of No t Flying . I have a feeling that organizations would 
be more receptive to evaluations if they proved the worthwhileness of 
the established programs. Evaluation in the service of program advo- 
cacy would be an effective tool for agencies seeking higher budgets 
and support for their preferred programs. Yet many of the evaluations 
undertaken during the past decade have cast serious doubts on the cost- 
effectiveness of major domestic programs. For every evaluation that 
highlights program success, there are handfuls which pronounce failure 
or uncertain results. This has been the sad fate of many of the housing 
and education innovations launched so enthusiastically in the 1960's. 
The market for evaluation dries up when evaluation becomes an anti-or- 
ganization weapon. 

Worse yet, evaluation has been concentrated on program innovations 
rather than on long established activities. Thus, Title I ESEA has been 
evaluated in dozens of communities from dozens of perspectives while the 
basic education programs which claim tens of billions of dollars have es- 
caped serious scrutiny. This means that the most venturesome, riskiest, 
activities are evaluated while, safe and perhaps unproductive programs 
are not subjected to the same tests. Strangulation by evaluation has 
been the dismal fate of many of the exciting education innovations of 
recent years such as performance* contracting and school vouchers. If 
evaluation is to succeed, it must be applied to old and new programs and 
must not be deployed as a weapon against program change. An anti-in- 
novation bias was inherent in the congressional budget reform bill for- 
mulated in 1973 by the Senate Government Operations Committee. As re- 
ported, new programs could not be implemented until they had undergone 



347 



mandatory pilot testing and evaluation. This bias was removed in 
later versions of the legislation. 

When an agency wants a new program, it might try to disregard 
or discredit an adverse evaluation. Such was the case in the early 
1970' s when the Defense Department promulgated a "fly before you buy 1 
policy for major weapons procurement. But shortly after a prototype 
F-14 crashed, the Navy rushed to order a substantial number of the 
new planes. 

(6) Inadequate Congressional Demand for Evaluation . In view of 
the powerful disincentives within executive agencies, the pressure 
for evaluation must come — if at all — from outside sources. Yet the 
weak agency commitment to evaluation is reinforced by lack of con- 
gressional interest. Congress rarely insists that agencies system- 
atically monitor their own performance and report the findings at 
hearings or other legislative forums. At least two explanations may 
be offered for congressional disinterest: first, committees may 
function as program advocates as much as program overseers; second 
Congress tends to be more oriented to prospective legislation than 
to past enactments. 

Authorizing committees may become advocates for the agencies 
and programs they are charged to oversee. Sometimes this is due 
to the "capture" of the committee by the agency, a process which is 
fomented by close and continuing contact between agency and commit- 
tee staffs and by the understandable inclination of congressmen to 
identify with the missions and activities of the agencies under 
their purview. Still another factor is that the authorizing role 
often casts the committee in an adversary relationship, vis a vis 



348 



the appropriations committees with the result that the authorizers 
feel impelled to support the agencies under their jurisdiction. 

As a legislating institution, Congress tends to rate its success 
and effectiveness in terms of the bills it passes rather than in terms 
of agency fidelity to earlier legislation. Congress is oriented to 
the future, not to the past, so that there is a chronic neglect of 
its oversight role. 

The Role of Congress 

Yet oversight is not a new role for Congress but one fully in 
accord with its generic purpose as a check on executive authority. 
With the growth of big Government, and in particular big bureaucracy, 
the importance of this role is greater than ever before. Inadequate 
attention to oversight cannot be blamed on inadequate staffing, though 
there surely remain critical staff shortages in certain areas. The 
evidence, however, is that the enlargement of staff and other re- 
sources during the past decade has not yet produced a breakthrough 
with regard to oversight by Congress. In comparing congressional 
capability versus practice, one is reminded of the epitaph on the 
tombstone of an athiest, "All dressed up and nowhere to go." If Con- 
gress is on the brink of a revised interest in legislative oversight, 
part of the reason is that for the first time it is tooled up to per- 
form this role in an effective manner. 

Some of the recent steps to bolster the oversight function have 
been mentioned by earlier speakers and these include the designation 
of oversight subcommittees in the House, expansion of GAO's role in 
program evaluation, and the new congressional budget process. None 



349 



of these laudable moves ensures the success of legislative oversight, 
but they at least remove some of the impediments to this role. 

It should be noted that I have been using the term "program over- 
sight" in referring to executive branch activities and "legislative 
oversight" when referring to Congress. This distinction is predicated 
on a division of labor between the two branches, with executive agen- 
cies carrying the brunt of evaluation efforts and Congress overseeing 
agency performance. There is not much sense in trying to make Con- 
gress into an evaluating institution, though on occasion Congress may 
perform this function. Evaluation should not be a sometime activity, 
conducted according to the inevitable spasms of legislative interest, 
but a regular part of administrative implementation of congress ionally- 
authorized programs. The appropriate role of Congress is to monitor 
performance and to prod agencies to evaluate their own activities. To- 
day a 1946 "continuous watchfulness" remains the essence of legisla- 
tive oversight. Perhaps the biggest payoff from a more independent 
and vigorous oversight role would be to stimulate more and better 
evaluations by executive agencies. Consequently, in the concluding 
portion of my talk, I will identify some ways in which Congress can 
goad agencies to perform useful evaluations. 

(1) Identification of Objectives . Despite all the emphasis on 
early legislative specification of objectives and intent, there is a 
limit as to how much precision and clarity can be achieved in com- 
plicated and controversial legislation. Legislators often are pro- 
voked by a sense of a problem, not by some firm objective. They 
perceive something awry, are moved to legislative's remedy, but do 
not have a precise notion of what the future should be, except that 



350 



they want the problem eliminated. When legislation is controversial 
and objectives clash, there is a tendency to paper over differences 
by being vague about purposes and intent. Thus, parties with differing 
objectives are able to subscribe to the same legislative package. 

When Congress muddles through without a clear specification of 
purpose, the process of evaluation is not aborted. Rather, it must 
begin with executive implementation rather than with textbook clarity 
about objectives. Congress, in fact, has two legitimate tasks to get 
the Executive to produce relevant evaluations. First, Congress can 
write into law a mandate for the agency to evaluate its program; sec- 
ond, Congress can demand that the evaluative measures used by the agen- 
cy reflect congressional interest and perspectives, not merely the 
orientation of the implementing agency. Statutory set asides of funds 
for evaluation do not suffice to establish an adequate evaluation pro- 
cedure. What is required is the prescription in law of a process that 
agencies must adhere to in evaluating their activities, including 
milestones for the crucial events in the process, and a reporting 
schedule. 

(2) Keep Evaluations Honest . If self-evaluation is difficult, one 
should not be surprised by the encroachment of organizational bias into 
even sophisticated evaluations. One form of bias arises out of the self- 
protection practiced by agencies to avert unfavorable findings. The 
best antidote for this kind of mischief is to "let it all hang out," 
assumptions and all, so that the judicious reader can determine the 
validity of the results. A more formidable kind of bias arises out 
of differing executive and legislative perspectives, in particular the 
holistic viewpoint of administrators versus the specific interests of 



351 



congressmen. Congress might be more concerned about the distributional 
effects of a program — who gets what — than about the impact on net na- 
tional welfare. However, congressional interest in program impact on 
specific target groups might not be reflected in the design of an eval- 
uation unless there is adequate legislative input at the start. One of 
the functions of the new oversight subcommittees, therefore, might be 
to examine prospective evaluations to gauge their sensitivity to the 
distribution of costs and benefits among various groups. 

(3) Expand the Time Framework . The time perspective of most eval- 
uations is notoriously short and there is real danger that the pace of 
legislative politics might induce agencies to work with even narrower 
time spans. Typically, an education experiment is tested during the 
ten months of the school year and if results are not forthcoming, the 
experiment is terminated. The demand is for quick results, to over- 
come within a few months the handicaps of years and generations. It 
is not surprising that so many evaluations show failure for programs 
inaugurated with promise and fanfare only months earlier. 

Patience is very much in short supply in program evaluation, not 
only because longitudinal studies are costly and difficult to engineer 
but also because administrators and legislators have a "now" orienta- 
tion and are unwilling to forego immediate gain in the quest for fu- 
ture improvement . 

At the present time, the technology of social evaluation is far 
more advanced than our understanding of basic social processes. We 
are bewildered when some Title I evaluations show success while others 
point only to failure, as if there were random or unknown factors at 
work. The real problem is that we have been willing to invest in 



352 



social experimentation but not in social understanding. At this 
juncture, it might be wiser to commission long-term studies than 
to finance more quickie research. But unless Congress is willing 
to tolerate the long view, agencies will continue to engage in LIFO- 
type evaluations, with the last program started, the first one ter- 
minated. 

(4) Make Them Readable . Many of the recent methodological gains 
in the state of the art have contributed to an escalation in the com- 
plexity and obscurantism of evaluations. Perhaps the most difficult 
task in evaluation is to translate quality research into readable 
policy analysis. The analyst often is insensitive to the importance 
of this task or incapable of escaping the constraints of his adopted 
language, but Congress need not suffer incomprehension gladly. Con- 
gress should insist that the fruits of evaluation be communicated in 
a straightforward and understandable way, that jargon and sophistry 
not be made into barriers to sound public policy. A side benefit 
might be to expose faulty assumptions, inadequate data, and question- 
able interpretations. When an evaluation is chock full of mumble 
jumble, the advantage is all on the side of the analyst, and Congress 
cannot exercise much independent judgment. Readable evaluations are 
indispensable if evaluation is not to become an executive violation 
of legislative power and independence. 

(5) Input into the Legislative Process . There is no special 
value in evaluation; the payoff comes in using evaluative findings 
for program design and implementation. Yet, there often is little 
follow-up to an evaluation; once done, the case is closed and the 
evaluators move on to other matters. Each evaluation is regarded 



353 



as a discrete, special event rather than as part of an overall 
policy process. 

Congress has a vital role in encouraging the application of 
analysis to public policy. Its two key points of input are the 
authorization and appropriations processes, and ray expectation is 
that neither presents a perfectly hospitable environment at the 
present time. As discussed earlier, the impulse for authorizers 
to double as program advocates impairs their ability to function 
as program evaluators . With regard to the appropriations process, 
it seems as targetted on incremental patterns as executive branch 
budgeting. Yet there is some possibility that the new congressional 
budget process might bolster the incentives for evaluation at these 
pivotal legislative stages, if only because of the pressure to con- 
sider particular spending demands in the light of overall national 
priorities and other claims on the budget. Moreover, the new con- 
gressional process might contribute to a narrowing of the authori- 
zations-appropriations gap, the proclivity to authorize one level 
of expenditure but to appropriate at a much lower level. More 
realistic authorizations might encourage committees to take more 
careful looks at the programs subject to their oversight. 

As long as each evaluation is an "event," a special occasion 
that commands special notice and applause, we can be sure that the 
actual progress of evaluation in terms of its utilization for pub- 
lic choice is inadequate. Only when evaluation becomes a regular 
part of program implementation will it recede as something special 
and become only something useful for Government. By this test, we 
all have a long way to go. 



355 



APPENDIX A 

JK 1015 c Union Calendar No. 31 

94th Congress, 1st Session .-..--. House Report No. 94-61 



OVERSIGHT PLANS OF THE COMMITTEES 

OF THE 

U.S. HOUSE OF REPRESENTATIVES 



A REPORT 

BY Tin: 

COMMITTEE ON GOVERNMENT 
OPERATIONS 




March 14, 1975. — Committed to the Committee of the Whole House 
on the State of the Union and ordered to be printed 



U.S. GOVERNMENT PRINTING OFFICE 
88-008 WASHINGTON : 1975 



Reproduced by the Library of Congress, Congressional Research 
Service. Ifareh 31, 1975. 



356 



PADL N. McCLOSKEY, Jr., California 
SAM STEIGER, Arizona 
GARRY BROWN, Michigan 
CHARLES THONE, Nebraska 
ALAN STEELMAN, Texas 
JOEL PRITCHARD, Washington 
EDWIN B. FORSYTHE, New Jersey 
ROBERT W. KASTEN, Jr., Wisconsin 
WILLIS D. GRADISON, Jr., Ohio 



COMMITTEE ON GOVERNMENT OPERATIONS 
JACK BROOKS, Texas, Chairman 

L. H. FOUNTAIN, North Carolina FRANK HORTON, New York 

JOHN E. MOSS, California JOHN N. ERLENBORN, Illinois 

DANTE B. FASCELL, Florida JOHN W. WYDLER, New York 

TORBERT H. MACDONALD, Massachusetts CLARENCE J. BROWN, Ohio 

WILLIAM S. MOORHEAD, Pennsylvania GILBERT GUDE, Maryland 

WM. J. RANDALL, Missouri 

BENJAMIN S. ROSENTHAL, New York 

JIM WRIGHT, Texas 

FERNAND J. ST GERMAIN, Rhode Island 

FLOYD V. HICKS, Washington 

DON FUQUA, Florida 

JOHN CONYERS, Jr., Michigan 

BELLA S. ABZL'G. New York 

JAMES V. STANTON, Ohio 

LEO J. RYAN. California 

i'ARDISS COLLINS, Illinois 

JOHN L. BURTON. California 

RICHARDSON I'REYER. North Carolina 

MICHAEL HARRINGTON, Massachusetts 

ROBERT F. DRINAN, Massachusetts 

HI (WARD MEZVINSKY. Iowa 

BARBARA JORDAN, Texas 

GLENN ENGLISH, Oklahoma 

ELLIOTT H. LEVITAS, Georgia 

DAVID W. EVANS. Indiana 

ANTHONY MOFFETT, Connecticut 

ANDREW MAGUIRE, New Jersey 

LES ASl'IN, Wisconsin 

William M. Jones, General Counsel 

John E. Moore, Staff Administrator 

William H. Copenhaver, Associate Counsel 

Lynne Higgineotham, Clerk 

J. P. Carlson, Minority Counsel 

(ID 



357 
LETTER OF TRANSMITTAL 



March 14, 1975. 
Hon. Carl Albert, 

Speaker of the House of Representatives, 
Washington D.C. 

Dear Mr. Speaker: Pursuant to rule 18(b) of the Committee on 
Government Operations, I submit herewith the committee's report to 
the 94th Congress, as required by House rule X, 2(c) relating to the 
oversight plans of the standing committees of the House of 
Representatives. 

Jack Brooks, Chairman. 



358 
CONTENTS 



Pane 

Introduction 3 

Background J? ZIIZI""I"~ """ """ 5 

Oversight coordination ~_ ~_~~~~~ 7 

Committee responses 9 

Committee on Agriculture 13 

Committee on Appropriations 14 

Committee on Armed Services 21 

Committee on Banking, Currency and Housing 24 

Committee on the Budget 29 

Committee on the District of Columbia 31 

Committee on Education and Labor 32 

Committee on Foreign Affairs 37 

Committee on Government Operations 40 

Committee on House Administration 50 

Committee on Interior and Insular Affairs 52 

Committee on Interstate and Foreign Commerce 54 

Committee on the Judiciary 57 

Committee on Merchant Marine and Fisheries 62 

Committee on Post Office and Civil Service 65 

Committee on Public Works and Transportation 68 

Committee on Rules 70 

Committee on Science and Technology 72 

Committee on Small Business 76 

Committee on Standards of Official Conduct 78 

Committee on Veterans' Affairs 79 

Committee on Ways and Means 80 

. (V) 



359 



Union Calendar No. 31 

94th Congress ) HOUSE OF REPBESENTATIVES J Report 

1st Session J \ No. 94-61 



OVERSIGHT PLANS OF THE COMMITTEES 

OF THE 

U.S. HOUSE OF REPRESENTATIVES 



14, 1975.— Committed to the Committee of the Whole House on the 
State of the Union and ordered to be printed 



Mr. Brooks, from the Committee on Government Operations, 
submitted the following 

REPORT 



71-867 O - 76 - 24 



360 



OVERSIGHT PLANS OF THE COMMITTEES OF THE 
U.S. HOUSE OF REPRESENTATIVES 



Introduction 



In adopting the Committee Reform Amendments of 1974, the 93d 
Congress recognized there were serious shortcomings in what is one 
of the most important congressional functions — oversight of Federal 
programs and agencies. 

The requirement first set forth in the Legislative Reorganization 
Act of 1946 that each standing committee exercise a continuing study 
and review of the statutes under its jurisdiction and the executive 
agencies and departments responsible for administering them, was 
made far more explicit by the 93d Congress. 

In a further effort to strengthen the overall oversight function of 
the House, it included in the reform amendments a provision now in- 
corporated in clause 2(c) of rule X, which reads as follows: 

At the beginning of each Congress an appropriate repre- 
sentative of the Committee on Government Operations shall 
meet with appropriate representatives of each of the other 
committees of the House to discuss the oversight plans of such 
committees and to assist in coordinating all of the oversight 
activities of the House during such Congress. Within 60 days 
after the Congress convenes, the Committee on Government 
Operations shall report to the House the results of such meet- 
ings and discussions and any recommendations which it may 
have to assure the most effective coordination of such activi- 
ties and otherwise achieve the objectives of this clause. 
(3) 



361 



Background 

As one of the weights Congress can drop on the scale in an effort to 
achieve a balance between the legislative and executive branches, over- 
sight has assumed an increasingly important role in recent years. This 
trend reflects a widespread concern in Congress that the growth of the 
Federal Establishment has outstripped congressional efforts to super- 
vise it. The chilling example of unchecked Federal power afforded by 
"Watergate-related issues should, alone, provide a sufficient spur for 
action. 

But, oversight should not be seen only as a club held^over the execu- 
tive branch. Ideally, it should complement executive management, pro- 
viding clear policy limits within which an administrator is free to act. 
The certain knowledge that failures of administration will be taken 
into account by an alert oversight committee should produce the effi- 
ciency of operation that both Congress and the executive branch desire. 

Oversight, especially in recent years, has come to be viewed primar- 
ily as an investigative activity, and it is true investigations have 
played a significant role in helping Congress keep track of executive 
branch operations. But there are other ways of accomplishing the 
same end. some more suitable to the continuing supervision that is 
contemplated by the new oversight rules. Where the goal is to improve 
efficiency, reinforce policy guidelines or reduce costs, rather than to 
expose some flagrant misdeed in the executive branch, a conference 
between committee and agency representatives can frequently achieve 
the desired result. The authorization and appropriation process offers 
committees a periodic opportunity to review the administration of 
many departments and take whatever corrective action is needed. 

Oversight, in fact, is not a structured procedure in the manner of 
Congress legislative function. A continuously watchful eye is the main 
requirement for determining whether an executive agency is perform- 
ing a? it should, backed by a willingness to act when it is not. 

Although practiced in a variety of ways throughout the history of 
the nation, oversight was not formally authorized as a function of each 
standing committee until passage of the Legislative Reorganization 
Act of 1046. That same act revived the long defunct Committee on 
Expenditures in the Executive Department and gave it broad powers 
to investigate all Federal agencies and departments. 

In 1052. that committee was renamed the Committee on Government 
Operations and now it has been given an even broader mandate and 
asked to help fashion a system for a coordinated review of the Federal 
Government by all committees. 

The all-embracing nature of the Committee on Government Opera- 
tions' oversight jurisdiction, which empowers it to examine "the over- 
all economy and efficiency of government operations and activities.*' 
is underscored by the special oversight provisions of the Committee 



362 



6 



Heform Amendments of 1974. Clause 4(c) (2) of rule X states that the 
committee "may at any time conduct investigations of any matter 
without regard to * * * provisions * * * conferring jurisdiction 
over such matter upon another standing committee.'' 

The House rules also instruct the other standing committees to in- 
clude in their legislative reports a summary of any oversight findings 
and recommendations the Committee on Government Operations may 
have made under its broad authority to investigate all levels of the 
Government. 

By including these provisions in the rules, the House has again 
emphasized its acceptance of the principle of maintaining an oversight 
committee separate and apart from the committees that have responsi- 
bility for initiating and continuing legislative programs. While tiiese 
legislative committees carry out their oversight responsibilities in their 
assigned areas, only the Government Operations Committee has the 
total view of the overall governmental process and avoids any ques- 
tion of lack of objectivity in connection with its investigations. 

The Committee on Government Operations intends to use its broad 
grant of investigative power carefully, but believes that it is a valuable 
and essential tool in carrying out the oversight activities of the House. 



363 



Oversight Coordination 

The Committee on Government Operations recognizes the need for 
the various committees to coordinate their oversight activities and 
accepts its responsibilities under rule X, clause 2(c) to assist them in 
doing so. At the same time, the committee feels constrained to point 
out that it has neither the authority nor the desire to police the other 
committees of the House in the performance of their legitimate over- 
sight duties. 

Each standing committee has a clearly assigned jurisdiction over 
which it is authorized by clause 2(b) (1) of rule X to exercise general 
oversight. Some overlapping of jurisdictions is unavoidable, and the 
Committee on Government Operations does not read into clause 2(c) 
any authority for it to arbitrate jurisdictional disputes that may arise. 
Nor does it interpret the clause to give it any power to supervise the 
other committees to see if they are meeting their oversight responsi- 
bilities as outlined in accordance with the provisions of clause 2(c). 

The committee believes that the objectives of the provision concern- 
ing oversight coordination can be met by — 

(1) Getting each of the committees to focus on its oversight 
responsibilities and develop some plana for carrying them out 
early in each Congress; 

(2) Indicating where areas of mutual interest or duplication 
of effort between committees may occur , and 

(3) Spotting areas in which no committee has indicated plans 
for conducting oversight activities. 

The committee recognizes the difficulties and uncertainties involved 
in any effort to set down an oversight agenda covering the next 2 years. 
It is emphasized that the oversight plans included in this report are 
only the best estimate at this time of what the committees of the House 
will try to accomplish in the 94th Congress. 

The committee is particularly aware of the difficulties at the begin- 
ning of the 94th Congress when each of the committees of the House 
has a substantial number of new Members and where the committees 
are operating under the Committee Reform Amendments of 1974 
for the first time. At the time this report was prepared, a number of 
the committees had not yet completed all of their organizational proc- 
esses, and this presented them with particular problems in attempting 
to predict their program for the entire Congress. The Government 
Operations Committee appreciates the complete cooperation and assist- 
ance given it by each of the committees in the preparation of this 
report. 

Under the procedure the Committee on Government Operations 
elected to follow, its representatives met separately with those of each 
of the other 21 standing committees, discussed the requirements :>nd 
purposes of clause 2(c), and asked them to submit a statement setting 
forth both their general oversight jurisdiction and their specific plans 

(7) 



364 



s 



for this Congress. It is the belief of the Committee on Government 
Operations that the presentation of each committee's plan in its own 
words gives the House the clearest picture of what the committees 
see as their major oversight activities, free from any distortions or 
inadvertent misrepresentation that might result from efforts to inter- 
pret or evaluate the plans. 

As will be noted from reading the committee responses, several com- 
mittees claim oversight responsibility for the same general subject 
area. This is entirely consistent with the legislative responsibilities of 
these committees, but it is the hope of the Committee on Government 
Operations that all committees, through an examination of this report, 
will acquaint themselves with the oversight plans of the other com- 
mittees, identify areas of potential duplication, and take steps to keep 
such duplication to a minimum. 

One method of accomplishing this might be for committees sharing 
jurisdiction to notify each other when any oversight activity is to be 
undertaken and to share any information they may have gathered 
about the operations of a program. Investigations might be conducted 
jointly, staff resources might be pooled, requests for information from 
executive agencies coordinated, and the services of the General Ac- 
counting Office used jointly. 

Such steps could be especially effective in the area of factfinding, as 
distinct from matters of law or policy. For example, the subject of 
sugar pricing is of legitimate interest to a number of committees be- 
cause of its effect on foreign trade, agriculture and the consumer. Each 
committee with jurisdiction in these areas would have a different 
reason for examining the subject, but all might be served by a single 
set of facts dealing with supply, suppliers, and prices. 

In implementing for the first time these new oversight rules, the 
committee is relying heavily on the other standing committees to rec- 
ognize and meet their responsibilities. It may be that in future years 
the House will want to delineate more specifically the role it wants the 
Committee on Government Operations to play in this regard. At this 
point, however, it is the committee's hope that by getting the 21 stand- 
ing committees to focus on oversight early in each Congress and put 
down their plans for reviewing the programs under their jurisdiction, 
it has helped set the House on the course charted by the Committee 
Reform Amendments cf 1974. 

It is clear that the main purpose behind the adoption of clause 2(c) 
of rule X is to foster cooperation between the committees of the House 
in carrying out their oversight duties. With a conscientious effort by 
all committees, that highly desirable goal can be reached. 



365 



Committee Responses 

The oversight plans of the committees form an impressive catalogue 
of programs due to come under congressional scrutiny. It must be rec- 
ognized, however, that plans are all they are. Oversight, by its nature, 
is often a response to the unusual and the unforeseen. The unsettled 
state of the economy, the complex efforts to draft a national energy 
program, and the volatile world political-military situation, all could 
lead to executive actions that will require prompt exercise by Congress 
of its oversight responsibility. In its discussions with the other com- 
mittees, the Committee on Government Operations made it clear it 
does not expect them to be held to the plans outlined in this report. 
They must remain flexible, able to shift their oversight resources 
wherever they may be needed. 

As has been mentioned previously, each committee was requested to 
provide for the Government Operations Committee a statement of its 
plans in writing, over the signature of its chairman. These responses 
will inevitably vary in style and detail as each committee was urged 
to express in its own way its views of its responsibilities and its plans 
for carrying them out. In each case, however, the committee repre- 
sentatives were encouraged to state: (1) The structure of the com- 
mittee for performing its oversight functions; (2) a description of 
its continuing general oversight activities; and (3) and specific plans 
at this time for concentrating on particular programs under its 
jurisdiction. 

The committees were asked to make their responses brief in hopes of 
obtaining more widespread readership of the final document. The com- 
mittee recognizes that if the purpose of this rule is to be accomplished, 
the report of the committee must have widespread appeal and be of 
such form it can be used as a guide by large numbers of persons with 
varied interests including Members of Congress, congressional staffs, 
the media, and the public. 

The responses of the committees follow : 
(9) 



366 



RESPONSES OF THE COMMITTEES 



367 
COMMITTEE ON AGRICULTURE 



UJS. HorsE of Representatives, 

Committee on* Acricultukfv 
Washington, D.C., February 25, 1975.- 
Hon. Jack Brooks, 

Chairman, Government Operations Committee, House of Representa- 
tives, Washington, D.C. 

My Dear Mr. Chairman : This is in reference to the oversight activi- 
ties contemplated by the Committee on Agriculture during the 94th 
Congress. 

Pursuant to House Rule X, clause 2(c) and your letter to me, mem- 
bers of our respective Committee staffs met last week to survey the 
scope of this Committee's activities during the current Congress. 

This Committee intends to conduct oversight and discharge its in- 
vestigative responsibilities in several ways. 

First, we plan to rely on the membership of the Subcommittee on 
Department Operations, Investigation and Oversight for the bulk of 
this work. The Committee has in fact provided in its rules that none 
of our other nine Subcommittees shall undertake a major investigation, 
without the consent of the Chairman of the full Committee. 

Second, all of our Subcommittees will conduct oversight and review- 
in the consideration of legislative proposals under consideration. 

And finally, the full Committee will continue to exercise oversight 
and review over those agencies which are affected by measures conr- 
sidered in the full Committee. 

The areas we know will be examined in the 94th Congress include' 
the Food Stamp Program, for which we are now planning a thorough 
review, sugar pricing systems, crop disaster and the administration 
and operation of the Federal Fungicide, Rodenticide and Insecticide 
Act, and the target price and loan system on major agricultural crops. 

Naturally we cannot anticipate all the areas of upcoming oversight 
as events yet to unfold will dictate that course of action for us. Thus 
we feel that we are neither restricted nor committed to any specific 
oversight activities at this particular time. 

In order to give your Committee and the House a better picture of 
our oversight activities, I have attached a copy of our Report to the 
House for the 93rd Congress on the subject (House Report 93-1654).* 
I would draw special attention to the jurisdictional range and the 44 
specific oversight activities we carried out in 1973-74 as set forth on 
pages 4, 5, 6, 13 and 14. 

I hope this will be helpful to you in carrying out your duties under 
House Rule X. 
Sincerely, 

Thomas S. Foley, Chairman. 

•Document available In flies of the Committee on Government Operations. 
(13) 



368 
COMMITTEE ON APPROPRIATIONS 



House of Representatives, 
Committee on Appropriations, 
Washington, D.C., February 27, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, House of Repre- 
sentatices, Washington, D.C. 

Dear Mr. Chairman: I am writing in response to your letter of 
"February 13 and your request for information concerning the over- 
sight activities of the Appropriations Committee in the 94th Congress 
as required by the new provisions of Clause 2(c) of Rule X of the 
House. 

As you know, the Committee on Appropriations is specifically ex- 
empted from general legislative oversight responsibilities under 
Clause 2(b)(1) of Rule X. The Committee does, however, retain 
the fiscal oversight' function under Clause 2(b) {?>) of Rule X which 
was formerly contained in Clause 2(b) of Rule XI. Under this provi- 
sion the Committee is authorized to conduct studies and examinations 
of the executive departments and agencies as it deems necessary to 
assist it in deliberations on matters within its jurisdiction. 

As I understand the intent of the new rule, it is the responsibility of 
the Committee on Government Operations to perform a coordinating 
function with respect to the oversight activities of the various standi 
ing committees of the House and make such recommendations as your 
Committee may deem appropriate to promote the overall effectiveness 
of reviewing the ongoing activities of the federal establishment. It is 
understandable that the rule applies differently to the Appropriations 
Committee than to the legislative committees. 

The principal function of the Appropriations Committee under 
Rule X of the House is the "Appropriations of the revenue for the sup- 
port of the Government". In executing its responsibilities, the Com- 
mittee through its thirteen subcommittees, annually examines the 
budget estimates of federal organizations contained in the Budget 
of the United States. The vast majority of this activity is directed 
at determining what amounts of money will be required to execute the. 
various federal programs during the forthcoming year imder existing 
legislation. The Rules of the House prohibit the appropriation of 
monies for purposes not previously authorized. Therefore, the Com- 
mittee examination of budget requests typically incorporates the pro- 
cedure of reviewing the execution of the ongoing programs as an 
important yardstick in measuring appropriation requirements for the 
subsequent fiscal year. This process has substantial "oversight" charac- 

(14) 



369 



15 

t eristics. It would be difficult, if not impossible, to identify what por- 
tion of the Committee's annual budget examination might be classified 
as "oversight" in nature, but it is considerable. 

During the last session of Congress, the Committee conducted over 
400 days of hearings, generating some TO volumes of testimony and 
supporting material aggregating some 75,000 printed pages. For 
each of the last two years, nine of the thirteen regular annual appro- 
priation bills were passed through the House by the end of June. In 
each of these years three bills were delayed due to the lack of legisla- 
tive authorization. This year we again hope to clear as many of the 
appropriation bills as possible through the House by the end of 
June contingent upon the availability of the necessary authorizing 
legislation. 

The Committee has now begun hearings on the fiscal 1976 budget. 
1 hiring the hearings this year, the Committee will explore certain pro- 
grams and situations which have been the subject of special studies by 
the Committee's Surveys and Investigation Staff. The Committee has 
over a number of years delegated the power to originate requests for 
investigations to its subcommittees, w T ith the concurrence of the Chair- 
man and Kanking Minority Member of the full Committee required to 
implement such requests. The various subcommittees utilize the capa- 
bility of the Surveys and Investigations Staff to make examinations of 
specific, subjects. 

As a matter of information, the Surveys and Investigations StalF 
is headed by a permanent professional staff member wdio is assisted 
by three high-caliber professional personnel from the Federal Bureau 
of Investigation under an arrangement dating back to 1943. These men 
are obtained from the F.B.I, and are placed directly on the payroll of 
the Appropriations Committee for the duration of their tour. Each 
serves two years in an apprenticeship capacity and then one year as a 
senior man. After this three-year assignment is completed, the men re- 
turn to the Bureau to continue their professional careers. 

When an investigation is ordered, experts in the field are assembled 
on an ad hoc basis, from outside or from within the government, to 
conduct the prescribed study and prepare a report thereon for the use 
of the Committee. The Committee delegates to this staff the responsi- 
bility for the selection of individual experts, the criteria being com- 
petence and objectivity. 

Once a particular study is completed, the investigator returns to his 
parent organization which is appropriately reimbursed for his time. 
Since the investigators are not permanent personnel, the system is elas- 
tic and economical. It is a unique arrangement insofar as we know, and 
the Committee has been well satisfied with the unbiased, objective re- 
ports rendered. 

During the 93rd Congress, the Investigative Staff conducted 95 stud- 
ies concerning various agencies* operations and programs. During 
this two-year period. 134 people on loan from 18 agencies of the Gov- 
ernment spent 1.489 man-months engaged in the conduct of studies and 
the preparation of reports. This is equivalent to about 124 man-years. 
Some of these studies, together with studies initiated this year, will lie 
an important part of the Committee's activities this year which should 
be classified as "oversight'" in nature. There follows a listing of studies 



370 



16 



•and investigations which have been initiated as a result of the hearings 
last year or in anticipation of hearings for fiscal year 1976 and repre- 
sent subjects which are candidates for "oversight" attention this year. 

A rt.tt.nl I n»- rati 

Name of study 

A detailed review of the operations of the Defense 
Investigative Service of the Department of Defense. 

Programs and activities of the National Security Agency- 
Utilization of military facilities — closure of McCoy Air 
Force Base. 

Phasedown of the Naval Air Station, Alameda. Calif 

Phasedown of the Hamilton Air Force Base. Calif 

Phasedown of Ft. Wainwright, Fairbanks, Alaska 

Closure of the Naval Shipyard, Hunters Point. Calif 

A review of the navigation systems of the Federal Gov- 
ernment. 

Air traffic control systems in the Department of Trans- 
portation and the Department of Defense. 

Proposed relocation of the Fleet Missile Systems Anal- 
ysis and Evaluation Group of the Department of the 
Navy. 

A review of the worldwide operations of the Defense 
Mapping Agency. 

Property maintenance, repair, and minor constructions 
in the Department of Defense. 

Budgeting and operational problems of the Federal 
Buildings Fund of the General Services Adminis- 
tration. 
"The use of automatic data processing and communica- 
tions systems of the Food and Drug Administration. 
Analysis of the operational costs at Howard University 
compared with educational institutions of comparable 
size. 

A review of the operation and activities of the Minority 
Business Enterprise program of the Department of 
Commerce. 

A review of the Community Mental Health Centers' 
staffing grants program supported by the Department 
of Health, Education, and Welfare. 

A review of the Maternal and Child Health formula 
grant program of the Department of Health, Educa- 
tion, and Welfare. 

Analysis of the overhead reimbursement policy of the 
Department of Health, Education, and Welfare. 

Analysis of bed occupancies and credits to research 
grants under the General Clinical Research Centers 
program of the National Institutes of Health. 

A review of the evaluation programs of the Department 
of Health, Education, and Welfare. 

A study of Federal programs which support the purchase 
of instructional equipment by institutions of higher 
learning — Office of Education of the Department of 
Health, Education, and Welfare. 

Analysis of the manpower. management policy and prac- 
tices in the Department of Health, Education, and 
Welfare. 

A review and analysis of the various problems associated 
with the construction of the Gathright Lake, Va., 
flood control project, Army Corps of Engineers. 
A review of the contracts and grants awarded by the Of- 
fice of Education for the support and improvement of 
postsecondary education — Department of Health, Ed- 
ucation, and Welfare. 



Actual (or estimated) 
date of completion 
Dec. 16, 1974. 

(Mar. 7. 1975.) 
June 24, 1974. 

Aug. 30, 1974. 
Sept. 24, 1974. 
Sept. 24, 1974. 
Sept. 27, 1974. 
Feb. 12, 1975. 

(Mar. 14, 1975.) 

Nov. 4, 1974. 



(Mar. 7, 1975.) 
(Mar. 19, 1975.) 
Feb. 19, 1975. 

Feb. 10. 1975. 
(Mar. 14, 1975.) 

(Mar. 6, 1975.) 

(Mar. 19, 1975.) 

Feb. 18, 1975. 

(Mar. 24, 1975.) 
Jan. 10, 1975. 

Feb. 25, 1975. 
(Mar. 3, 1975.) 

Feb. 13, 1975. 
Feb. 3, 1975. 
(Mar. 12, 1975.) 



371 



17 



Same of ttudy 

An analysis of headquarters staffing and activities of the 
various components of the Department of Defense. 

A review and analysis of the activities of the Consumer 
Product Safety Commission. 

An overall review of the activities of the Federal Trade 
Commission. 

A specific analysis of the information systems and auto- 
matic data processing activities of the Federal Trade 
Commission. 

An overall review of the activities of the Food and Drug 
Administration. 

A general review of the F-14 aircraft program of the 
Department of the Navy, including delivery schedule, 
advance payments, performance of the aircraft, com- 
parative cost of F-14 and F-15, and cost effectiveness 
compared with possible replacement aircraft. 

A review of the activities of the National Center 
for Toxicological Research of the Food and Drug 
Administration. 

A review of the requirements and criteria used by the 
military services in providing graduate degrees to mili- 
tary personnel. 

A report to the Committee on the factors which have 
contributed to the rising cost of electricity to the 
American consumer and the effect Government regula- 
tions have had on such increases. 

A report of the overseas deployment of tactical nuclear 
weapons of the Department of Defense. 

An analysis of the Migrant Health Program of the De- 
partment of Health, Education, and Welfare. 

A review of the airborne warning and control system 
proposed by the Department of the Air Force. 

An inquiry into the operation of the Tokota Air Base, 
Japan, and the consolidation of military facilities in 
the Kanto Plain area. 

The feasibility of adopting the unit replacement concept 
for armed forces deployed overseas. 

A detailed analysis of the computer operations of the 
Energy Research and Development Administration. 

A report on the cost overruns associated with the vari- 
ous programs of the Atomic Energy Commission (En- 
ergy Research and Development Administration). 

A comprehensive analysis of the New Communities As- 
sistance Program of the Department of Housing and 
Urban Development. 

Analysis of the scientific research programs of the De- 
partment of Agriculture. 

An analysis of the regional and area office structure of 
the Department of Housing and Urban Development. 

An analysis of the need for and cost of the Rocket As- 
sisted Projectile Program of the Department of the 
Army. 

An analysis of the research facilities of the Environ- 
mental Protection Agency. 

A study of the restructuring of the Naval Reserve, to- 
gether with an evaluation of the impact such restruc- 
turing has had upon the training, readiness, morale, 
etc. of the personnel involved. 

An analysis of the need for a proposed second source for 
hull and turret castings for the M48 tank — Depart- 
ment of the Army. 



Actual (or mtimated) 
date of completion 
(Mar. 27, 1975.) 

(Mar. 11, 1975.) 

Feb. 7, 1975. 

Jan. 10, 1975. 

Feb. 26, 1975. 
(May 15, 1975.) 

(Mar. 31, 1975) 
(Apr. 18, 1975.) 
(Mar. 14, 1975.) 

(May 30, 1975.) 
Feb. 20, 1975. 
(Apr. 11, 1975.) 
(June 13, 1975.) 

(May 26, 1975.) 
(Mar. 31, 1975.) 
(Mar. 26, 1975.) 

(Mar. 31, 1975.) 

(Mar. 14, 1975.) 
(Mar. 21, 1975.) 
(Apr. 2, 1975.) 

(Mar. 21, 1975.) 
(Apr. 21, 1975.) 

(Feb. 4, 1975.) 



372 



18 

Actual (or estimated) 
Xame of study date of completion 

A detailed review of the construction programs of the (May 1, 1975. t 

Veterans' Administration, including an analysis of cost 

overruns associated therewith. 
A detailed analysis of the XJI-1 tank program of the (July 11, 1975.) 

Department of the Army. 
A review of the compliance of the Department of the (Mar. 21, 1975.) 

Interior with certain language contained in the House 

Report for fiscal year 1975 concerning the Intermoun- 

tain Indian School and the National Indian Training 

Center at Brigham City, Utah. 

"With respect to the above listing, it is important to note that because 
the studios originate with the subcommittees, any information devel- 
oped during the course of an investigation is reported to the chair- 
man of the subcommittee which requested such study or examination 
as well as the Chairman and Ranking Minority Member of the full 
Committee, and such information may be released for publication 
only when the subcommittee so determines. This procedure is in- 
corporated in Section 8 of the Committee's rules. 

The above listing should not be interpreted as the full extent of the 
Committee's oversight activities. Requests for investigations are initi- 
ated by the various subcommittees from time to time as, in the judg- 
ment of their members, are needed to illuminate a particular appro- 
priations issue or problem related to a department or agency within 
their respective jurisdictions. It would not be practical to undertake 
to identify what further subjects will receive attention as might be 
generated by the regular annual review process which has just begun. 

In addition to the information made available to the Committee 
through its Surveys and Investigations Staff, its "oversight"' func- 
tions are buttressed by the continual utilization of the resources of the 
General Accounting Office. The Committee receives a copy of every 
GAO report sent to the Congress. In recent years the scope of the 
GAO auditing and review capability has been enlarged to include 
management surveys. For some seventeen years, the GAO has ex- 
tracted from these reports and compiled in separate volumes a list of 
so-called ''significant Audit Findings'' for special use by the Commit- 
tee staff in the annual appropriations hearings. This compilation is 
designed to identify problem areas in an individual agency which 
might have applicability to other organizations. These findings relate 
to matters which are felt to require corrective action by the Commit- 
tee's efforts, through legislation, or through administrative efforts. 
You and your staff may be interested in reviewing the latest 191 
page report which was issued by the General Accounting Office on 
January 2, 1975. 

The extent of use of GAO findings by the Appropriations Com- 
mittee is reflected in the number of references to the GAO in the an- 
nual appropriation hearings on the House and Senate in recent years 
as follows : 

Year : 

1968 649 

1969 727 

1970 701 

1971 722 

1972 722 

1973 632 



373 



19 



Additionally, the Committee frequently calls upon GAO to make 
special studies and investigations. During FY 1974 GAO prepared 
10 studies of significant programs as requested by the Committee as 
well as .53 staff studies on defense weapons systems. 

Pending for possible attention in connection with the fiscal year 
1976 hearings are GAO studies in a number of areas. Selected examples 
are as follows : 

Request for views on section 1015 of Public Law 93-344 with 
regard to budget rescissions and deferrals 

Comprehensive review of commissary operation and commissary 
stores operated by the military services 

Review all funds obligated under the Presidential Transition 
Act 

Army's program to modernize ammunition plants — Depart- 
ment of the Army (continuing work) . B-172707 
International shipments of household goods — DOD 
Move policies related to unaccompanied tours of duty 
"Do It Yourself" programs for movement of household goods 
Planning and budgeting for new test facilities at Arnold En- 
gineering Development Center, Tennessee 

Department of Defense Requirement for Helicopters 
Feasibility of Consolidating Chaplain Schools 
Training Rates for Pilots and Navigators 

Legality of crediting reimbursements to the appropriation year 
collected. Also, requests GAO in conjunction with OMB and 
Treasury Department, reexamine the appropriation and appor- 
tionment processes to determine what legislative and regulatory 
changes may be needed to eliminate the existing potential for 
the M accounts and related Surplus Funds accounts to be used to 
increase fund availability and obligational authority to avoid 
possible overobligation or overexpenditure violations under the 
Ant i -Deficiency Act 

Study allegations that the Army Electronic Command at Fort 
Monmouth. New Jersev, has overobligated funds available to it 
for FY 1974 by some $56 million 

Review directed by Conference Report on FY DOD Appro- 
priations. House Report Number 93-1363, dated 9/18/74— DOD*s 
stockpiling of war materials or equipment for use by foreign 
countries 

Review of enlisted separation payments (Directed by House 
Report No. 93-1255, Fiscal Year 1975 Department of Defense 
Appropriation Bill) 

Loan Guarantee Programs under the Rail Passenger Service 
Act and the Regional Rail Reorganization Act 

Cost comparisons of subsidized housing provided under the 
Section 23. Section 236, and revised Section 23 programs, and 
monitor and keep Congress advised of costs of the new revised 
Section 23 program as it is being developed and implemented. 
In 1972 GAO was reorganized on a functional basis. This was de- 
signed to encourage better response to demands for broad-based and 
program-type reviews. It provided a vehicle for senior staff to achieve 
a high degree of specialization in the subject matters of functional 
areas assigned to them. 



374 



20 

Further, an Office of Program and Budget Analysis was established 
in 1975 in response to the Budget and Impoundment Control Act 
of 1974. The specific responsibility of this office, to the Appropriations 
Committee and others, includes the following : 

1. Budget issue identification and analysis, 

•2. Macro analysis of major budget and fiscal issues, 

3. Budget related revenue analyses, 

4. Analysis of executive budget and supporting data, 

5. Coordination of GAO-wide support of the Congressional 
budget process. 

6. Impoundment review and reporting. 

7. Liaison with Congressional Budget Office and the Con- 
gressional Budget Committees and liaison with other Congres- 
sional Committees. 

The Committee is presently contemplating how best to further and 
better utilize the GAO in regard to their traditional services and 
expanding capabilities but especially with respect to securing rapid 
reports on topics which relate to issues arising in current appropria- 
tions hearings. It is felt that such a capability may be especially fruit- 
ful in the defense field. 

"While I realize that some of my comments in regard to the "over- 
sight v plans of the Appropriations Committee have been somewhat 
general, I hope the specifics that I have indicated are sufficient to 
illustrate the scope and thrust of the work we undertake on an annual 
basis and will be useful to you in developing the report which must 
be rendered in accordance with Clause 2 of Rule X. If there is any 
other information which we might provide, we want to cooperate to 
the fullest extent. 

Best regards. 
Sincerely, 

George Mahon, Chairman. 



375 
COMMITTEE ON ARMED SERVICES 



U.S. House of Representatives, 

Committee on Armed Services, 
Washington, D.C., February 28, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, 
U.S. Ilouse of Representatives, Washington, D.C. 

Dear Mr. Chairman : This will acknowledge your letter of Feb- 
ruary 13, 1975 with reference to the provisions of clause 2(c) of 
Rule X of the rules of the House for the 94th Congress requiring 
coordination of the oversight activities of the standing Committees 
of the House by the Committee on Government Operations. We have 
consulted with members of your staff and submit the following report 
of oversight contemplated by the Armed Services Committee during 
the 94th Congress. 

This Committee will not utilize a special oversight subcommittee 
to carry out its oversight responsibilities. Rather, as has been the case- 
over the years, the standing legislative subcommittees and the Special 
Subcommittee on Intelligence will be responsible for the oversight 
function in their areas of jurisdiction. 

The general oversight responsibilities of the House Armed Services 
Committee, as contained in Rule X of the rules of the House of 
Representatives, are as follows: 

(1) Common defense generally. 

(2) The Department of Defense generally, including the Depart- 
ments of the Army, Navy, and Air Force generally. 

(3) Ammunition depots: forts; arsenals; Army, Navy, and Air 
Force reservations and establishments. 

(4) Conservation, development, and use of naval petroleum and of 
shale reserves. 

(5) Pay. promotion, retirement, and other benefits and privileges 
of members of the armed forces. 

(6) Scientific research and development in support of the armed 
services. 

(7) Selective service. 

(8) Size and composition of the Army, Navy, and Air Force. 

(9) Soldiers' and sailors' homes. 

(10) Strategic and critical materials necessary for the common- 
defense. 

Note. — Under Rule X 3(a) the Committee has a special oversight 
function with respect to international arms control and disarmament 
and military dependents education. 
(21) 



ri-867 O - 7S - 25 



376 



22 

In response to the House rules and your specific request, the follow- 
ill*; areas and subjects have been designated for special oversight at- 
tention during the 94th Congress : 

(1) Intelligence 

To cover the Central Intelligence Agency, the National Security 
Agency and Department of Defense and service intelligence com- 
ponents, including intelligence activities in the United States and 
abroad and the collection, analysis and dissemination of intelligence 
information, intelligence estimates and covert activities. Also, in- 
cluded are those areas where such departments or agencies participate 
in intelligence matters outside their respective immediate chains of 
command. The overview will also include personnel policies and prac- 
tices, relevant procurement policies and practices and budget matters. 

(2) Energy 

The exploration, development and any limited production in the 
Naval Petroleum and Oil Shale Reserves with particular reference 
to the maintenance of a defense petroleum asset. 

(3) Foreign Military Sales and Their Impact on Readiness of 
U.S. Forces 

(4) U.S. Training of Forei£rn Military Personnel 

(.">) Cost-Effectiveness Studies of tlie Military Departments 

(6) Foreign Investment in U.S. Defense-Oriented Corporations 

(7) Department of Defense "World-Wide Command and Con- 
trol System 

(8) Thefts and Losses of Military Weapons 
(0) Military Commissary Stores 

(10) Strategic and Critical Materials in the National Stockpile 

(11) Military Real Estate Acquisitions and Disposals 

(12) The Naval Base at Diego Garcia 

(13) T 1 e Airborne Command Post Program 

(14) The Manned Bomber in the Strategic Arsenal 
(lft) The Planned Production Contract for AW ACS 

(16) Naval Vessels and Shipyards 

Special attention will be given to an examination of the procure- 
ment and construction of naval vessels and maintenance and produc- 
tion problems in public and private shipyards. 

(17) Anti-Submarine Warfare 

This overview will include procurement and construction of anti- 
submarine warfare systems. 

(15) Transfer and Disposal of Naval Vessels 
(10) Sealift in our National Security 

(20) Department of Defense Personnel (Military and Civilian) 
Particular attention will be focused on the all-volunteer force con- 
cept. tli° total force concept, retention of military personnel, military 
medical care delivery and the CHAMPUS program, dependent poli- 
cies in the U.S. and overseas and the Uniform Code of Military 
Justice. 

(21^ Military Compensation. Promotion and Retirement 
Attention will be focused on proposals which involve revision of 
the total spectrum of laws relating to officer selection, promotion and 
separation procedures, as well as modernization of the military retire- 
ment >vstem. Also, there will be a reexamination of the Survivor 



377 



23 

Benefit Plan for military personnel. In addition, there will be a major 
review of the total range of military pay and allowances, including 
the implementation of special pay legislation for professionals in the 
health care fields, personnel in military flight status and other areas 
of special pav categories. 

(22) the Armv Tank Program 

(23) U.S. Military Commitment to NATO 

(24) The Soviet Threat 

(25) The Military Situation in the Middle East 

(26) Arms Control and Disarmament 

( 27 ) The Selective Service System 

(28) Research and Development 

General legislative oversight incorporating all of the research, de- 
velopment, test and evaluation programs of the individual service de- 
partments and defense agencies, with special emphasis on those pro- 
grams which are being considered for ultimate procurement, such as 
the AWACS; 15—1 ; the XM-1 tank program; various missile pro- 
grams and support systems; the TRIDENT program; military air- 
lift : independent research and development. The Committee also in- 
tends to review the following subject matter in connection with its 
RDT&E oversight responsibilities: military requirements and mili- 
tary support technology for chemical weapons ; development of greater 
standardization and commonality in military weapons systems; the 
implications of detente on RDT&E effort ; the Strategic Arms Limita- 
tion Treaty; and various other foreign policy implications in the 
RDT&E programs of the military departments. 

I believe it is evident, Mr. Chairman, that the foregoing informa- 
tion is not restrictive in any way since experience indicates that future 
circumstances may require additional oversight emphasis in areas not 
previously marked for special attention. In such instances it may well 
be necessary to divert the Committee resources in new directions at 
the expense of an area or areas previously marked for special atten- 
tion. Also, in submitting the information above concerning intelligence 
oversight, we are mindful of the special requirements placed upon the 
Select Committee on Intelligence established by H. Res. 138. In addi- 
tion, it should be pointed out that the Committee conducts continuous 
oversight activities through correspondence and other communications 
into all facets of the provisions of Titles 10 (Armed Forces). 37 (Pay 
and Allowances), 50 and 50 Appendix (War and National Defense), 
United States Code by virtue of the unique wav such statutes cut 
across the daily operations and activities in the Department of De- 
fense and in some instances, the Central Intelligence Agency. Finally, 
in carrying out its oversight responsibility the Committee on Govern- 
ment Operations may care to have reference to the Report of Activi- 
ties of the Committee on Armed Services, 93rd Congress, House Report 
93-1662 of January 2, 1975. 

With best regards, I am, 
Sincerely, 

Melvin Price, Chairman. 



378 
COMMITTEE ON BANKING, CURRENCY AND HOUSING 



U.S. House of Representatives, 
Committee on Banking, Currency and Housing, 

Washington, D.C., February 28, 1975. 
Hon. Jack Brooks, 

Chairman, Government Operations Committee, House of Representa- 
tives, Washington, D.C. 
Dear Mr. Chairman : In reference to your recent letter and discus- 
sions between members of your staff and the staff of the House Com- 
mittee on Banking, Currency and Housing, the following information 
is submitted to you concerning the oversight activities of our 
Committee. 

The House Committee on Banking, Currency and Housing has estab- 
lished a Subcommittee on General Oversight and Renegotiation. The 
jurisdiction of this Subcommittee, as set forth in the Committee Rules, 
states as follows: 

The General Oversight and Renegotiation Subcommittee shall 
assist the House Committee on Banking, Currency and Housing 
in appraising the administration of the laws and regulations under 
the jurisdiction of the Committee and present such recommenda- 
tions as deemed necessary to the appropriate subcommittee (s) 
of the Committee. 

Further, the Subcommittee shall exercise continuing oversight 
of the execution by the administrative agencies concerning any of 
the laws the subject matter of which reside within the jurisdiction 
of the Committee and shall study all pertinent reports, documents 
and data pertinent to the jurisdiction of the Committee and make 
the necessary recommendations or reports thereon to the appropri- 
ate subcommittee (s) of the Committee. 
Further, the Subcommittee shall have full jurisdiction over the 
Renegotiation Act of 1951, as amended. 

Further, as stated in Rule IX of the Rules of the Committee on 
Banking, Currency and Housing, the Oversight Subcommittee is 
charged with responsibilities as set forth in Attachment 1. It is under- 
stood, as set forth in this rule, that the Oversight Subcommittee of the 
House Committee on Banking, Currency and Housing shall not in- 
fringe upon the jurisdiction of another subcommittee of the full Com- 
mittee without proper consultation. The Oversight Subcommittee, 
therefore, is directed to coordinate and cooperate with the legislative 
subcommittees of the full Committee. The Oversight Subcommittee is 
precluded from investigating any subject matter falling within the 
jurisdiction of a legislative subcommittee of the Committee unless the 
legislative" subcommittee is in accord with the undertaking. 

(24) 



379 



25 



The Chairman of the Committee and the Committee is to be in- 
formed of any proposed oversight undertakings by any subcommittee. 
If any question arises concerning the jurisdiction of the Oversight 
Subcommittee and a legislative subcommittee, the Chairman of the 
Committee is charged with the responsibility of arbitrating the dif- 
ferences. If necessary, the matter can be appealed to the Democratic 
Caucus and, if necessary, the full Committee. 

Some of the oversight and investigative activities of the various 
subcommittees of the House Committee on Banking, Currency and 
Housing that will be undertaken this year are set forth, by subcom- 
mittee, in Attachment 2. This list of proposed oversight and investi- 
gative activities of the various subcommittees of the full Committee 
may not be all-inclusive. Instances may arise requesting immediate 
attention by the committee or one of its subcommittees which have 
not been anticipated. Further, the amount of time devoted to a given 
investigation may exceed the initial estimated time and, therefore, it 
may not be possible to carry out oversight in all of the areas detailed 
above. 

Mr. Chairman, I believe the above statements answers m detail 
your questions concerning this Committee's oversight structure and 
functioning. 

Sincerely, 

Hent?y S. Keuss, Chairman. 

[Attachment 1] 

Rule No. IX 

OVERSIGHT subcommittee 

^a) In order to assist the House in: 

(1) its analysis, appraisal, and evaluation of (A) the applica- 
tion, administration, execution, and effectiveness of the laws 
enacted by the Congress, or (B) conditions and circumstances 
which may indicate the necessity or desirability of enacting new 
or additional legislation, and 

(2) its formulation, consideration, and enactment of such mod- 
ifications or changes in those laws, and of such additional legis- 
lation, as may be necessary or appropriate, there shall be estab- 
lished in conformity with Rule XV an Oversight Subcommittee. 

(b) The Oversight Subcommittee shall review and study, on a con- 
tinuing basis, the application, administration, execution, and effective- 
ness of those laws, or parts of laws, the subject matter of which is 
within the jurisdiction of the committee, and the organization and 
operation of the Federal agencies and entities having responsibilities 
in or for the administration and execution thereof, in order to deter- 
mine whether such laws and the programs thereunder are being imple- 
mented and carried out in accordance with the intent of the Congress 
and whether such programs should be continued, curtailed, or elimi- 
nated. In addition, the Oversight Subcommittee shall review and 
study any conditions or circumstances which may indicate the necessity 
or desirability of enacting new or additional legislation within the 
jursdiction of the committee (whether or not any bill or resolution 



380 



26 

has been introduced with respect thereto), and shall on a continuing 
basis undertake futures research and forecasting on matters within 
the jurisdiction of the committee. The Oversight Subcommittee shall 
in no way limit the responsibility of the subcommittees from carrying 
out their oversight responsibilities. 

(c) The Oversight Subcommittee shall review and study on a con- 
tinuing basis the impact or probable impact of tax policies affecting 
subjects within the jurisdiction of the committee. 

DOMESTIC MONETARY POLICY SUBCOMMITTEE 

This subcommittee in the immediate future will begin hearings on 
legislation which would require interest to be paid on funds owned by 
the Federal Government residing in non-interest bearing accounts, 
primarily in commercial banks. This has been a matter which has been 
under investigation for a number of years, and finally the Treasury 
Department has agreed that such funds should bear interest. Estimates 
indicate the Federal Government could realize upwards of $400 million 
a year from such legislation. 

Investigative undertakings of the Subcommittee this year will in- 
clude inquiry into: 

The necessity and desirability for monetary policy to be made an in- 
tegral part of the overall economic policy-making process now carried 
on by the Executive Branch and the Congress, thus making monetary 
policy more directly accountable to the political process. 

The relationship between the monetary policy responsibilities of the 
Federal Reserve and its bank regulatory responsibilities, including the 
questions of whether a serious conflict of interest exists in having the 
same government agency carrying out at the same time both monetary 
and regulatory responsibilities; and the extent to which the Federal 
Reserve uses its regulatory power to achieve its monetary policy goals. 

To what extent does the Federal Reserve use its authority to allocate 
credit on an ad hoc basis while publicly opposing Congressional at- 
tempts to establish a fair, open and rational system of credit allocation 
with specifically established criteria based on Congress' perception of 
the public interest. 

The effect of monetary policy and other financial actions on interest 
rates, allocation of credit, and the structure and functioning of do- 
mestic and foreign financial institutions, including such possible effects 
as creating increased concentration, a lessening of competition, and 
restricting adequate access to credit to various elements in the economy. 

HOUSING AND COMMUNITY DEVELOPMENT SUBCOMMITTEE 

Investigative oversight activities of the Subcommittee on Housing 
and Community Development this year will include : 

Oversight hearings on the New Communities Program and on 
HUD administration and suspension of new communities program. 

Mortgage credit hearings and studies to provide a comprehensive 
review of the role and operations of the FHA, FNMA, GNMA, 
FHLMC, Farmers' Home Administration, the private mortgage in- 
surance industry, and State and local credit programs. 



381 



27 

Section 8 Housing Subsidy and Community Development Block 
Grant Programs, oversight of HUD implementation of programs 
and evaluation of performance under programs. 

701 Comprehensive Planning Program; A-95 Review Process, 
examination of program benefits and administration of programs. 

Preservation of the existing housing stock, study of the progress 
and procedures for the preservation of older housing and the con- 
tribution of the Section 8 and Community Development Programs. 

ECONOMIC STABILIZATION' SUBCOMMITTEE 

The Economic Stabilization Subcommittee will devote a great deal 
of its time to oversight inquiries into the operations of the Defense 
Production Act. This inquiry will explore the procedures and proc- 
esses whereby under the Defense Production Act, loans and loan 
guarantees can be granted to businesses for the development of tech- 
nological processes and for the exploration, development and mining 
of strategic metals and materials. This, obviously, could have some 
positive implications for energy resource development. In addition, 
oversight investigations in this area will include the operations of 
the 1 >efense Production Act, regarding stockpiling provisions for na- 
tional defense purposes — both the acquisition and sale thereof. Ob- 
viously, too, the stockpile provisions of the Defense Production Act 
could be used for stockpiling of energy resources. 

FINANCIAL INSTITUTIONS SUPERVISION, REGULATION AND INSURANCE 
SUBCOMMITTEE 

This Subcommittee intends to conduct investigative oversight hear- 
ings into the reasons for the recent bank failures, which hopefully 
will ultimately provide the Committee with the necessary ingredients 
needed to revive and revamp the financial institutions federal regula- 
tory structure. 

This Subcommittee will, in addition, continue to investigate red- 
lining, whereby access to credit sources in central cities in viable 
neighborhoods is denied credit due to reasons which are not economic. 
Also, the Subcommittee will investigate the question of conversion of 
mutual savings and loan associations into stock associations and re- 
view existing cease and desist authority for bank and savings and 
loan holding companies. 

HISTORIC PRESERVATION AND COINAGE SUBCOMMITTEE 

The Subcommittee on Historic Preservation and Coinage, under 
the Chairmanship of the Honorable Robert G. Stephens, Jr., will look 
into the coinage programs of the United States Government as they 
are being developed as part of our Bicentennial celebration. 

In addition, this Subcommittee, again in anticipation of the Bicen- 
tennial celebrations will conduct a complete review of Title I of the 
National Housing Act as it relates to the funding of studies, acquisi- 
tions and/or preservation of historic structures in the United States. 



382 



28 

INTERNATIONAL DEVELOPMENT INSTITUTIONS AND FINANCE 
SUBCOMMITTEE 

Investigative and oversight activities of the Subcommittee will com- 
prise a general review of U.S. participation in the various inter- 
national development institutions with a view to determining what 
the continuing role of the U.S. as a participant should be in the 
institutions. It has been some years since this subcommittee conducted 
a general review of the internal operations and lending policies of 
these institutions which will be done this year. 

GENERAL OVERSIGHT AND RENEGOTIATION SUBCOMMITTEE 

General oversight subjects which may be undertaken by this sub- 
committee, in consultation and accord with the substantive legislative 
subcommittees of the full committee, possibly would include bank 
closing. Federal Reserve System building programs, government tax 
and loan accounts, Federal crime insurance, military credit unions. 
Section 312 rehabilitation laws and laws of the several states affecting 
condominium purchases. 

CONSUMER AFFAIRS SUBCOMMITTEE 

Investigative activities of this Subcommittee will include a review 
of the effectiveness of the existing truth-in-lending legislation and in- 
vestigation into the practices of credit collecting agencies and some 
oversight hearings into specific consumer price increases. 

INTERNATIONAL TRADE, INVESTMENT AND MONETARY POLICY 
SUBCOMMITTEE 

The Subcommittee on International Trade, Investment and Mone- 
tary Policy, under the Chairmanship of the Honorable Thomas M. 
Rees, will review the structure, function and activity of the Export- 
Import Bank, especially as regards Ex-Im lending and guarantee 
programs and projects to Communist countries. 

In addition, the jurisdiction of this subcommittee extends to IMF 
U.S. policy regarding investments abroad, especially by multinational 
corporations and foreign investment in the United States. 

In addition, the Subcommittee will begin an overall investigation of 
operations and in this regard legislation will probably be transmitted 
to the Hill which would seek to create an IMF special oil facility. If so, 
this legislation, which would call for the participation of perhaps up 
to $25 billion by the United States and/or other industrial nations, 
would be considered by this Subcommittee. 



383 
COMMITTEE ON THE BUDGET 



U.S. House of Representatives, 

Committee on the Budget, 
Washington, D.C., March 3, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, U.S. House of 
Representatives, Washington, D.C. 

Dear Mr. Chairman : The purpose of this letter is to inform you of 
the Budget Committee's oversight responsibilities in general, the 
specific areas of oversight expected to be undertaken this year, and the 
structure to be utilized to carry out oversight activities. 

As you know, the Committee on the Budget only recently organized 
itself for the 91th Congress, and much of the Committee's time has 
been devoted to dealing with the extent to which the new budget 
process called for in the Congressional Budget and Impoundment 
Control Act of 1974 (Public Law 93-344) will be implemented this 
year. Consequently, since the Committee has made no formal deci- 
sions on specific oversight activities to be conducted this year, the 
following should be regarded as tentative in nature. We will, of course, 
inform you as soon as possible of specific oversight activities decided 
upon by the Committee. 

GENERAL OVERSIGHT RESPONSIBILITIES 

The Budget Committee is responsible for : 

(1) making continuing studies of the effect on budget outlays of 
relevant existing and proposed legislation and reporting the results of 
such studies to the House on a recurring basis ; 

(2) requesting and evaluating continuing studies of tax expendi- 
tures, devising methods of coordinating tax expenditures, policies, 
and programs with direct budget outlays, and reporting the results of 
such studies to the House on a recurring basis ; 

(3) reviewing, on a continuing basis, the conduct by the Congres- 
sional Budget Office of its functions and duties. (P.L. 93-344, section 
101.) 

In addition, P.L. 93-344 directs the Budget Committee to undertake 
a series of studies in specific areas : 

(1) not later than the close of the 95th Congress, the Budget Com- 
mittee is required to report to the House on procedures used under 
section 301 (b) concerning the enrollment of appropriation and entitle- 
ment legislation (section 301 (b) ) ; 

(2) on a continuing basis, the Budget Committee is required to study 
provisions of law which exempt certain Federal agencies from inclu- 
sion in the budget (section 606) ; and 

(29) 



384 



30 

(3) on a continuing basis, the Budget Committee is required to study 
proposals designed to improve and facilitate methods of congressional 
budget-making (section 703). 

SPECIFIC AREAS OF OVERSIGHT TO BE UNDERTAKEN IN 1975 

The Budget Committee has made no formal decisions concerning 
which of the studies enumerated above should be initiated this vear. 
All involve important matters which must receive continuing atten- 
tion and analysis. 

Preliminary work has begun in one area : a study of tax expendi- 
tures. The Committee recently asked the Joint Committee on Internal 
Revenue Taxation to analyze the estimates of tax expenditures made 
by the President in the FY 1976 budget, and heard testimony on tax 
expenditures on March 3 from Professor Stanley Surrey of Harvard 
University. A substantial amount of the Committee's oversight activ- 
ity this year may involve the tax expenditures area. 

The Committee will certainly give special attention this year to 
the initial activities of the Congressional Budget Office. This Office, 
established on February 27 upon appointment of a Director by the 
Speaker of the House and the President pro tempore of the Senate, will 
be organizing and staffing during the next few months so that it can 
provide the House and Senate Budget Committees with assistance 
during the latter part of 1975 and, more importantly, in 1976 when 
the new congressional budget process is fully implemented. Our Com- 
mittee will work closely with the new Office to assure that it is or- 
ganized and staffed in a manner designed to provide the highest quality 
of service to the Budget Committees, other committees of the House 
and Senate, and Members of Congress. 

STRICTURE TO BE USED TO CARRY OUT OVERSIGHT RESPONSIBILITIES 

The Committee has made no formal decision on the structure to be 
used to carry out its oversight functions. I feel that a series of task 
forces should be established to undertake the various studies enumer- 
ated above; and perhaps a special oversight task force should be 
established on the Congressional Budget Office. I plan to raise this 
matter with the Committee at an early date, and will report its deci- 
sions to you as soon as possible. 

I hope this information is helpful. Please let me know if I can be 
of any further assistance. 
Sincerely yours, 

Brock Adams, Chairman. 



385 
COMMITTEE ON THE DISTRICT OF COLUMBIA 



U.S. House or Representatives, 
Committee on the District of Columbia, 

Washington, D.C., February 21, 1975. 
Hon. Jack Brooks, 

Chairman of the House Committee on Government Operations, 2157 
It ay burn House Office Building, Washington, D.C. 
Dear Mr. Chairman: This comes in response to your letter of 
February 13, 1975, in which you requested a relatively concise listing 
of oversight activities contemplated by the District Committee during 
the 94th Congress. While the Committee has yet to determine how the 
oversight responsibilities will be divided between the full Committee 
and six (6) subcommittees, we are certain that the following areas will 
bo subject to investigative oversight by the Committee : 

1. The operation of District Government and the transition to home 
rule. 

2. St. Elizabeth's Hospital. 

3. The Bicentennial organization and activities in the National 
Capital Region. 

4. The taxicab industry in the District of Columbia. 

5. National capital regional issues, such as aspects of the Potomac 
River and the water and sewage system of the District. 

6. Social Service delivery systems. 

7. Implementation of the 1973 Governmental Reorganization plan 
with a view to its economy and efficiency. 

8. Operation of the National Capital Service Area. 

9. METRO with respect to its financial affairs and related matters. 

10. The administration of justice in the District of Columbia, in- 
cluding an investigation of the "preventive detention" issue. 

11. Economic development and housing in the city. 

The Chief Counsel of the Committee, Mr. Robert B. Washington, 
Jr., is the staff person designated for contact by members of your Com- 
mittee Staff. I, of course, will be happy to discuss any of these items 
with you, if you feel it would be helpful. 
Sincerely, 

Charles C. Diggs, Jr., 
Chairman, Committee on the District of Columbia. 
(31) 



386 



COMMITTEE ON EDUCATION AND LABOR 

House of Representatives, 
Committee on Education and Labor, 

Washington, D.C., March 3, 1975. 
Hon. Jack Brooks, 

Chairman* Committee on Government Operations, U.S. House of Rep- 
resentatives, Washington, D.C. 

Dear Mr. Chairman: Pursuant to your letter of February 13, rep- 
resentatives of this Committee conferred with your staff on February 
20 on our Committee's oversight activities for the coming Congress. 
This letter results from that meeting and is an attempt to assist you to 
carry out your responsibilities under Clause 2(c) of title X of the 
Rules of the House. 

At our organizational meeting, the Committee on Education and 
Labor elected to continue to impose oversight responsibility on each 
of our legislative subcommittees rather than establish a separate sub- 
con imittee which would attempt to do oversight in the many areas 
over which we haA-e jurisdiction. The balance of this letter will indi- 
cate what the general jurisdictional responsibilities for each subcom- 
mittee are and, as far as possible, lay out the present plans for specific 
oversight activities. 

These plans are necessarily tentative. Nor should they be regarded 
as exhaustive. Our Committee has always felt obliged to respond to 
crises or serious problems as they arise. For example, the Special Sub- 
committee on Labor in the 93d Congress watched very carefully the 
development of negotiations and the strike actions at the Brookside 
Mine, as it had the West Coast dock strike in the 92d Congress. Al- 
ready this year our Subcommittee on Labor Standards has sent a staff 
investigator to the State of Washington to investigate the impact of 
the Fair Labor Standards Act Amendments of 1974 on some of the 
federal employees at the Puget Sound Naval Shipyard. All of our 
subcommittees will continue to be prepared to respond to such problem 
situations as the need arises. The need for this sort of response, the 
development of new and presently unforeseen problems requiring at- 
tention, and our legislative duties may force us to curtail some of the 
projected activities. We will continue, however, as we have in years 
past to do as much and as effective oversight as is possible. 

Both your letter and your staff emphasized "concise listings" and 
"brevity". I hope the following meets your needs in this respect with 
fairness to our subcommittees. 

subcommittee on elementary, secondary and vocational education 

General 

This Subcommittee has jurisdiction over matters affecting education 
through the high school level and vocational education. 

(32) 



387 



33 

Specific 

(1) The Subcommittee will give careful scrutiny to the implementa- 
tion of the Education Amendments of 1974. Implementing regulations 
are being developed and submitted to the Congress for review and 
oversight has already begun. 

These amendments affect the Elementary and Secondary Education 
Act, the Adult Education Act, Indian education, impact aid, educa- 
tion of the handicapped and other related laws. 

(2) Hearings both in Washington and in the field are planned to 
look into Indian education. 

(3) There will be a continuation of oversight hearings and investiga- 
tion into the administration and effectiveness of the Vocational Edu- 
cation Act of 1963. 

Legislation is expected to result from these activities. 

SUBCOMMITTEE on labor-management relations 

General 

This Subcommittee has jurisdiction over relationships between em- 
ployers and employees and the federal laws, statutes and regulations 
relating to the collective bargaining process. 

Specific 

(1) The Subcommittee will continue oversight of the administra- 
tion of the Service Contract Act of 1965 with a view to bringing to 
light certain questions of interpretation and policy that have arisen. 

1 2) Oversight will continue into the administration of the National 
Labor Relations Act. 

(3) The appropriateness and effectiveness of the Consumer Price 
Index and its administration will be studied. 

(4) The Labor Department's administration of the Walsh-Healey 
Act will be the subject of oversight activity. 

SUBCOMMITTEE ON LABOR STANDARDS 

Genera? 

As its name implies, this Subcommittee has jurisdiction over wage 
and hour standards including welfare and pension plans and coal 
mine health and safety. 

Specific 

(1) The Subcommittee will study the impact and implementation 
of the Fair Labor Standards Amendments of 1974, more precisely the 
coverage of domestic service employees, public employees (federal, 
State and local) and the child labor provisions, particularly in 
agriculture. 

(2) The Subcommittee will continue its study of the impact of im- 
ports and exports on American employment and on the wage and 
hour standards affecting American business. Particular emphasis will 
be given to the automobile industry. 

(3) The administration of the Black Lung Benefits Program (title 
IV of the Federal Coal Mine Health and Safety Act of 1969) will be 
looked into. 



388 



34 

(4) The implementation and impact of the Employee Retirement 
Income Security Act of 1974 will require considerable and careful 
study which the Subcommittee expects to undertake. 

(5) The Subcommittee also hopes to give some attention to the re- 
lationships among the Davis-Bacon, "Walsh-Healey and Service Con- 
tract Acts. . 

(6) The operation and assistance by the United States of overseas 
schools attended by the dependents of U.S. officials and military per- 
sonnel stationed abroad will continue to be the subject of oversight 
activities. 

SUBCOMMITTEE OX MANPOWER, COMPENSATION, AND HEALTH 
AND S.YTETY 

General 

The Subcommittee has jurisdiction over the areas of occupational 
safety and health, compensation and manpower programs, including 
training, public service employment, unemployment and full employ- 
ment programs, special unemployment assistance and job placement. 

Specific 

The Subcommittee plans oversight and investigative activities in 
the following areas: 

(1) Manpower — the Subcommittee is alreadj' looking into the ad- 
ministration and implementation of the public service jobs program 
and the Comprehensive Employment and Training Act with a view 
to early legislation in this area. It will also conduct investigations into 
the WIN Program and the job placement provisions of the Wagner- 
Peyser Act. 

(2) Occupational safety and health. 

(3) Federal Employees Compensation Act. 

(4) The relationship of the Metallic and Nonmetallic Mine Safety 
Act to the Coal Mine Health and Safety Act and the Occupational 
Safety and Health Act will be studied. 

(5)" The operations of private employment agencies will be in- 
vestigated. 

(6) The Longshoremen's and Harbor Workers' Compensation Act. 

SUBCOMMITTEE ON SELECT EDUCATION 

General 

The Subcommittee has diverse jurisdiction over a number of special 
education programs including such things as education of the handi- 
capped, environmental and alcohol and drug abuse education. It also 
has jurisdiction over a number of programs providing services to the 
elderly, the handicapped, the needy, child development, and a variety 
of programs related to art, culture and libraries. 

Specific 

(1) The Subcommittee will investigate the applicability of space age- 
research and technology to the needs of handicapped persons. 

(2) The Subcommittee will investigate the administrative practices 
and responsibilities of national. State and local agencies in the admin- 
istration of social service programs under its jurisdiction. 



389 



35 

(3) The Subcommittee will continue the study of applying engineer- 
ing technology, the use of computers and media devices to a variety of 
educational programs. 

(4) The impact oi' federal tax laws on the handicapped, the elderly 
and families in need of child care will be studied. 

(5) The Subcommittee will study the quality of the social services 
being provided by the program under its jurisdiction and conduct 
a study in depth of the recipients of those services to develop a clear 
picture of who they are and whether the intended l>eneficiaries are 
receiving services as intended. 

SUBCOMMITTKE ON* POSTSECON'DARY EDUCATION* 

General 

The Subcommittee has jurisdiction over education beyond the high 
school level. 
Specific 

(1) The, Subcommittee will continue its major effort at oversight 
relating to the student aid provisions of the Higher Education Act 
which it expects will result in legislation in the first Session of this 
Congress to improve the operation of the Act, to ease the default 
problem in student lending and to reduce the degree to which present 
student financial aid policies encourage tuition increases. 

(2) The Subcommittee expects to conduct oversight hearings in 
furtherance of the Committee's functions under Section 431(d) of 
the General Education Provisions Act requiring all standards, rules, 
regulations and requirements of general applicability issued by the 
Commissioner of Education to be submitted to the Congress forty- 
five days prior to their effective date during which the Congress, hav- 
ing reviewed the regulations, may return them to the agency for 
revision. 

('.)) The Subcommittee intends, in addition, to examine student 
aid programs in a number of areas where it has secondary jurisdic- 
tion to ascertain possible divergences in federal policy toward assist- 
ance levels. Among the programs that may be looked into are the 
student aid programs of the Public Health Service, the Law Enforce- 
ment Assistance Administration, the Department of Transporta- 
tion, the Environmental Protection Agency, the Veterans' Admin- 
istration, the Bureau of Indian Affairs, the Department of State, the 
Maritime Commission, the Social Security Administration, the Na- 
tional Science Foundation, the National Endowment for the Human- 
ities and others. 

(4) Other possible areas of oversight investigation being con- 
sidered include consumer protection in postsecondary education and 
the protection of student athletes. 

SUBCOMMITTEE ON* EQUAL OPPORTUNITIES 

General 

The Subcommittee lias general responsibility for reviewing the 
administration and effectiveness of programs authorized under the 
Economic Opportunity Act; the Juvenile Justice and Delinquency 



390 



Prevention Act of 1974; title VII of the Civil Eights Act of 1964; 
and the Domestic Volunteer Services Act of 1973 as well as other 
matters related to poverty and equal employment opportunities. 

Specif c 

(1) The Subcommittee plans to oversee implementation of the 
Headstart. Economic Opportunity and Community Partnership Act 
of 1974. including the transfers of programs from the Community 
Services Administration; the effectiveness of the Community Eco- 
nomic Development Program; the provision of services to handi- 
capped children in the Headstart Program; and a study of the 
impact of the reduction of the federal share for community action 
programs. 

(2) The Subcommittee plans oversight on equal employment oppor- 
tunities in four general areas: (a) oversight of the Equal Employment 
Opportunity Commission, particularly its backlog; (b) coordination 
between federal agencies with employment discrimination enforce- 
ment responsibilities: (c) the effectiveness of the Office of Federal 
Contract Compliance in administering Executive Order 11246 barring 
employment discrimination by federal contractors; and (d) employ- 
ment discrimination in federal agencies. 

(#) The Subcommittee plans investigation and oversight also into 
the ACTION Agency's implementation of domestic volunteer pro- 
grams : the failure of the Law Enforcement Assistance Administration 
in the Department of Justice to implement the Juvenile Justice and 
Delinquency Prevention Act of 1974; and the impact of the nation's 
tax policies on the poor. 

SUBCOMMITTEE ON AGRICULTURAL LABOR 

(Jencral 

The Subcommittee has oversight responsibility for all laws which 
affect migrant and seasonal farmworkers and their dependents. 
Specif c 

(1) The Subcommittee plans to oversee the implementation of the 
recently enacted amendments to the Farm Labor Contractor Registra- 
tion Act of 196o by the Department of Labor. 

(2) The Sul>eommittee plans to review and oversee the policies and 
practices of the Department of Labor in its certification of the use of 
offshore agricultural workers and the impact of such certification on 
the agricultural labor market. 

(3) The Subcommittee plans to oversee the transfer from the De- 
partment of Labor and the implementation by the Community Serv- 
ices Administration of title IH-b. developmental programs for mi- 
grant and seasonal farmworkers (Headstart. Economic Opportunitv 
and Community Partnership Act of 1974). 

(4) The Subcommittee will review the impact and implementation 
of provisions of existing laws administered by the Department of 
Health. Education, and Welfare whieh affect the health and education 
of migrant and seasonal farmworkers. 

With kindest regards, 
Sincerely, 

Carl D. Perkixs. 



391 
COMMITTEE ON FOREIGN AFFAIRS 



ITorsK of Representatives, 
CoMMirm-: ox Foreign* Affairs, 
Washington^D.C:, February 27. Wj. 
TIon. Jack Brooks. 

('hainrwu. Committee on Government Operations, House of 
He preventatives. Washington, D.C. 
Dear Mr. Chairman : Attached is material coiKerniii'g the oversight 
activities of the Foreign Affairs Committee which is being provided 
to you in accord with Clause 2(c) of the Rules of the House for the 
04-th Congress. 

I hope you will hud this information responsive to your request 
as outlined by your staff. 
"With best wishes. I remain 
Sincerely yours, 

Thomas E. Moruax. Chairman. 

oversight activities of the foreign affairs committee. 
9 4tii coxoress 

The ovei-sight activities of the Foreign Affairs Committee in the 
!>4th Congress will extend over the broad range of international affairs 
of interest to or connected with the United States. These activities 
will l>e pursuant to the responsibilities assigned to the Committee 
under the House rules. Rule X. l(i) lists jurisdiction for the Com- 
mittee starting with "Regulations of the United States with Foreign 
Nations Generally" and includes 16 further categories. In addition, 
special oversight functions are provided in Clause 3(d) of Ride X. 

By subject grouping, legislation and situations under Committee 
oversight include: international relations; conduct of foreign policy; 
international agreements; international and regional security arrange- 
ments; interventions abroad; intelligence activities relating to foreign 
policy; diplomatic privileges and immunities; operations of foreign 
affairs agencies: foreign humanitarian, food, economic and military 
assistance: foreign loans: information, scientific, and cultural ex- 
change; arms control; U.S. property abroad: boundary activities; 
international organizations, conferences and congresses; protection of 
American citizens and business interests abroad: international eco- 
nomic policy: international financial and monetary organizations: 
international commodity agreements (other than sugar) and trade; 
international fishing agreements: customs administration: and inter- 
national education. 

The Committee performs its oversight through activities of the full 
Committee, a Subcommittee on Oversight, and nine other functional 
(37) 



392 



subcommittees, utilizing the Committee staff and special studies per- 
formed for the Coiumittee or its subcommittees by such organizations 
as the General Accounting Office, the Congressional Research Service 
and by special consultants. 

Specific oversight activities now underway or planned by the full 
Committee or Subcommittees include, review of: 

U.S. interests, policies, aid, and commitments relating to Southeast 
Asia; POW/MIA issues; Current U.S. role. U.S. commitments and 
peace efforts relating to the Middle East; Demilitarization in the 
Middle East; U.S. aims supply policies and activities of U.S. defense 
contractoi-s in the Persian Gulf area; Palestinian situation; Restric- 
tions on U.S. aid to Turkey; U.S. policy toward Portugal and impli- 
cations for NATO; Spanish base rights agreement; Panama Canal 
treaty negotiations; Europe, detente, and U.S. forces; American policy 
toward Ethiopia; Military assistance, sales and arms transfer pro- 
grams; Role of military 7 assistance advisory groups; Use of private 
contract employees for military training overseas; Intelligence activi- 
ties relating to foreign policv; Nuclear proliferation and compliance 
witli NPT; 

U.S. export of nuclear technology; SALT IT. MRFE, (XT), and 
oilier arms control and disarmament negotiations; Implications of 
new conventional weapon technology for national security and arms 
control; Interaction of science and technology with U.S. foreign 
policy; Executive agreements; Implementation of P.L. 93-148, the 
War Powers Resolution; Congress' role in foreign policy; Foreign 
A Hairs- related required reports to Congress; Advisory commissions 
in I he foreign policy field; State Department operations; Foreign 
Building operations; U.S. Information Agency operations; Peace 
Corps operations; A 01) A operations; Foreign service personnel; 
Foreign Disaster Assistance Programs; 

International Aid to the Sahel Region; U.S. aid to the drought- 
stricken nations of Africa; All) implementation of Congressional ly 
mandated "new directions'' in foreign aid policy; U.S. -supported 
population programs; Implementation of food production assistance; 
Follow-up to the Workl Food Conference; Foreign policy aspects of 
IMi ISO; Overseas food distribution under PL 4H0; Use of foreign 
currency proceeds derived from PL 480; Food policies of countries 
receiving U.S. food aid; Food problems of developing countries; Ac- 
cumulation and disposal of local currencies: Overseas housing guar- 
anly program; Operations of Overseas Private Investment Corpora- 
tion; Operations of Inspector General. Foreign Assistance; Inter- 
national narcotics traffic; Effectiveness of operations of U.N. systems 
and the I '.S. -performance in that connection; 

Operations of certain U.N. specialized agencies, including 
UNESCO; Human Rights, U.S. foreign policy related to human 
rights, and the U.X. Human Rights Commission; U.N. Environment 
Program; International Conferences, including the Law of the Sea 
Cnnfeieiice; Seabed resources; Operations of international Imnking 
institution* to which the U.S. contributes: Ex-Tm Rank operations; 
Impact of international monetary issues on the American economy: 
Pi rforniaiicc of the ( 'ouncil on International Economic Policy; Moni- 
toriii" of foreign invcslmenl in 11.2$. bv federal agencies, of American 



393 



39 



investments in foreign countries, and the impact on the U.S. economy; 
International codes of conduct for multinational corporations; Impact 
of cartels, monopolies and multinational corporations on the economy; 
International commodity agreements (except sugar); Progress of 
multilateral trade negotiations: U.S. government-business relations 
affecting the U.S. position in international trade and investment; 
Political and economic impact of the world oil crisis; 

Impact of the energy crisis on developing countries; International 
Energy Agejicy; $25 billion energy safety net; Non-military uses of 
nuclear energy abroad, including related IAEA activities; Adminis- 
tration of export controls; Status of settled and unsettled foreign 
claims ; Strategic export limitations and operations under trading with 
the enemv legislation; U.S. policy toward and relationships with 
South Africa; U.S.-Afrka producer-consumer relationships; L T .S. 
lM>licy toward Latin America and the Caribbean; Implementation of 
legislation authorizing the Tijuana flood control project; U.S.- 
Canadian arrangements for Great Lakes management; Impact of 
Canadian actions on U.S. telecommunications; Means for improving 
efficiency of U.S. -funded international broadcasting; Western hemi- 
sphere immigration; Customs administration; External i-elations of 
Puerto Rico and the Trust Territory of the Pacific Islands. 

The oversight activities listed in this presentation represent what 
is < ui Tenth- in progren or planned for 1075 and 1976. Further over- 
sight projects will be developed subsequently. Because of the uncer- 
tainties in the foreign affairs field, it is impossible to give a complete 
listing this earh in the 94th Congress or to give assurance that some 
plans may not change as a l-esult of future developments and the need 
to divert our manpower resources to more urgent projects. 



394 
COMMITTEE ON GOVERNMENT OPERATIONS 



Although it has some major legislative responsibilities, the primary 
function of the Committee on Government Operations, as provided in 
the House rules, is to "review and study, on a continuing basis, the 
operation of Government activities at all levels with a view to deter- 
mining their economy and efficiency." 

It carries out this responsibility through seven subcommittees, each 
with jurisdiction over specific agencies and departments. 

The Legislation and National Security Subcommittee is assigned 
responsibility for the Departments of Defense and State, the Central 
Intelligence Agency, the Arms Control and Disarmament Agency, the 
General Accounting Office, most functions of the Executive Office of 
the President. Selective Service, and the Renegotiation Board. 

The Intergovernmental Relations and Human Resources Subcom- 
mittee watches over the Department of Health. Education, and Wel- 
fare, most activities of the Department of Agriculture, the Commodity 
Credit Corporation. Farm Credit Administration, and Federal Crop 
Insurance Corporation : the National Science Foundation and all agen- 
cies and programs involving interrelationships between the Federal 
Government and other levels of government, of which the most im- 
portant is revenue sharing. 

The Conservation. Energy, and Natural Resources Subcommittee 
has responsibility for the Department of the Interior, the Corps of 
Engineers, the Forest Service, the Environmental Protection Agency 
and Council on Environmental Quality, and all agencies' energy re- 
search and development, including the Rural Electrification Admin- 
istration and Tennessee Valley Authority. 

The Government Activities and Transportation Subcommittee has 
jurisdiction over the Department of Transportation. Civil Aeronautics 
Board. Federal Maritime Commission. Interstate Commerce Commis- 
sion. National Transportation Safety Board. St. Lawrence Seaway 
Corporation, and National Aeronautics and Space Administration, as 
well as the General Services Administration. Office of Telecommuni- 
cations Policy, Federal Property Act. and Federal Property Council. 

The Coommerce, Consumer, and Monetary Affairs Subcommittee 
deals with the Departments of Treasury and Commerce, the Federal 
Trade Commission. Small Business Administration. Federal Reserve 
System. Council of Economic Advisers, and the many agencies in- 
volved in commerce, consumer, and monetary affairs. 

The Manpower and Housing Subcommittee has the Departments of 
Labor and Housing and Urban Development as its prime responsi- 
bility, but also oversees the Veterans' Administration. Civil Service 
Commission, the Office of Economic Opportunity, and several com- 
missions and boards that operate in the field of manpower and labor 
relations. 

The Government Information and Individual Rights Subcommittee 
checks on the Justice Department and the Federal courts, the Federal 

(40) 



395 



41 

Communications Commission, U.S. Postal Service, and statutes deal- 
ing with freedom of information and the right of privacy, and the 
ngencies that administer them. 

Each of these subcommittees has submitted its oversight plans for the 
04th Congress in accordance with Rule X, Clause 2(c), and I hereby 
submit them for inclusion in this report. 

Jack Brooks, 

Chairman. 



LEGISLATION AND NATIONAL SECURITY 8UBCOMJ 

Jurisdiction 

The subcommittee will maintain oversight of the functions and 
operations of the Department of Defense, State Department, Agency 
for International Development, Executive Office of the President, 
General Accounting Office, Selective Service System, Central Intelli- 
gence Agency, Renegotiation Board, and Arms Control and Dis- 
armament Agency. 
Department of Defense 

Transportation Data Systems and Other GAO Reports; Foreign 
Troop Training Contracts with Private Corporations; Illegal Con- 
tracts for Computer Systems; Recommendations of Commission on 
Government Procurement ; Proper and Efficient Use of Defense Man- 
power; Further Study of "Parts Standardization in Weapon Sys- 
tems"; Contracting-Out Policies; Cost Overruns in Defense Procure- 
ment. 

State- AID 

Organizational Deficiencies and Staff Morale; AID Policies in For- 
eign Assistance; Consolidation of Administrative Support Functions; 
Collection of Delinquent Debts. 

Intelligence 

Overlapping and Duplication Among Agencies. 
Other Areas 

Office of Federal Procurement Policy; Office of Management and 
Budget — Domestic Council. 

Jack Brooks, Chairman. 

intergovernmental relations and human resources subcommittee 

House of Representatives, 
Washington, D.C., March 7, 1976. 
Hon. Jack Brooks, 

Chairman, Committee on Government. Operations, 2157 Raybvm 
House Office Building. Washington, D.C. 
Dear Jack : In response to your request of February 24, I am en- 
closing a brief listing of the more significant oversight activities pres- 
ently contemplated tor the 94th Congress. Almost all of these matters 
are currently under active investigation. 

Other matters will no doubt be added to our oversight agenda 
later on the basis of events taking place during the next two years. 
Sincerely, 

L. H. Fountain, Chairman. 
Enclosure. 



4a 

. Qvevsight activities presently contemplated during the 94th Con- 
gress incmdc Investigation and/or study of: 

Department of Health, Education, and Welfare 

Procedures and resources utilized by the Department of Health, 
tiducition, and Welfare in the investigation of fraud and other pro- 
gram irregularities. 

Administration of Social Security programs, with particular em- 
phasis on problems involving the Supplemental Security Income 
(.SSI) program. 

Administration of the Aid to Families with Dependent Children 
(AFDC) program. 

Administration of the Medicare and Medicaid programs. 

Administration of the Guaranteed Student loan program and other 
student financial aid programs administered by the Office of 
Education. 

. Regulatory policies, procedures and activities of the Food and Drug 
Administration for assuring the safety and effectiveness of new drugs. 

Effectiveness of Federal programs intended to combat cancer and 
heart diseases. 

Possible irregularities in travel expenditures by officials of the 
Department of Health, Education, and Welfare. 

Department of Agriculture 

Rural housing operations of the Farmers Home Administration. 

Selected aspects of the regulation of commodity futures trading by 
the Commodity Exchange Authority and successor agencies. 

Selected aspects of the administration of the Food Stamp program. 

Activities of the Department of Agriculture relating to exports of 
agricultural commodities. 

Intergovernmental Relations 

The General Revenue Sharing program established by the State 
and Local Fiscal Assistance Act of 1972 (P.L. 92-512). 

conservation, energy, and natural resources subcommittee 

House of Representatives, 
Washington, D.C., March 3, 1,975. 
Hon. Jack Brooks, 

Chairman, Committee on, Government Operations, Room 2157, 
liayburn Building, Washington, D.C. 

Dear Mr. Chairman: This is in response to your request of Feb- 
ruary 24 for a relatively concise listing of oversight activities con- 
templated by the Conservation, Energy, and Natural Resources Sub- 
committee during the present Congress. 

All of the items listed, of course, are subject to approval and pri- 
ority determination by the Subcommittee when it holds its organiza- 
tional meeting this month. Your suggestions also are actively solicited. 

With kind regards. 
Sincerely, 

William S. Moorhead, Chairman. 



397 

43 

Jurisdiction 

The Subcommittee maintains oversight of the activities and opera- 
tions of the following agencies and departments : Environmental Pro- 
tection Agency, Council on Environmental Quality, Energy Research 
and Development Administration, Federal Energy Administration, 
Federal Power Commission, Nuclear Regulatory Commission, Rural 
Electrification Administration, Tennessee Valley Authority, Depart- 
ment of the Interior, Corps of Engineers, Forest Service, and Indian 
Claims Commission. 
Oversight 

Investigative program and priority order subject to approval by 
subcommittee. 

Federal involvement of the above agencies in particular and of the 
Executive branch in general in the following activities : 

1. Development of oil shale deposits and on-shore oil and gas leases ; 

2. Conversion of power generators to coal and other fuels; 

3. New energy sources offering potential major benefits in the near 
future ; 

4. Construction of new gasification plants utilizing coal, oil shale 
and garbage; 

5. Solid wastes, including conservation, environmental and energy 
aspects ; 

6. Assisting State and local governments to obtain needed informa- 
tion on new, high-benefit energy, conservation and environmental 
advances which they could adopt ; 

7. Duplication and overlapping of energy, conservation and en- 
vironmental operations; 

8. National Park concession practices (continue review initiated 
last session) ; 

9. Pass-through of higher energy costs to consumers by public 
utilities ; 

10. Power Rate Increases (Bureau of Reclamation, Central Valley 
Project, California) ; 

11. Enforcement of Federal Mine Health and Safety Laws; 

12. Garrison Diversion Unit, North Dakota ; 

13. Study of automotive fuel economy ; 

14. Administration of the Fish and Wildlife Coordination Act; 
and 

15. Other items yet to be determined by future events, GAO reports, 
Member requests and new developmente in past investigations. 

government acitvittes and transportation subcommittee 

House of Representatives, 
Washington, D.C., March 10, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, 2157 Raybum 
House Office Building, Washington, D.C. 
Dear Mr. Chairman : This is in response to your request of Febru- 
ary 24, 1975, for a listing of oversight activities contemplated by the 



398 



44 

•Government Activities and Transportation Subcommittee during the 
present Congress. 

The listed items are, of course, subject to approval and priority 
determination by the Subcommittee when it holds its organizational 
meeting. 

Sincerely, 

Wm. J. Randall, 
Chairman, Government Activities and 

Transportation Subcommittee. 
Enclosure. 

.Assigned jurisdiction 

Transportation Matters. — Department of Transportation and com- 
ponent agencies including the U.S. Coast Guard and Saint Lawrence 
•Seaway Development Corporation; Civil Aeronautics Board; Federal 
Maritime Commission; Interstate Commerce Commission; National 
Aeronautics and Space Administration; National Transportation 
Safety Board. 

Government Activities. — General Services Administration ; Federal 
Property Act; Federal Property Council; Office of Telecommunica- 
tions Policy. 
Proposed oversight activities 

1. Air transportation of hazardous cargo. 

2. Conflicting altitude-control responsibility during aircraft land- 
ing approach. 

3. Requirements for altitude monitoring avionic equipment in com- 
mercial aircraft. 

4. Review of certain procurement actions by the Department of 
Transportation and Federal Aviation Administration involving land- 
ing-aid equipment. 

5. Coast Guard search and rescue operations. 

6. Utilization, operation, and management of the St. Lawrence Sea- 
way. 

7. Interstate Commerce Commission responsiveness to rate-change 
requirements in the public sector. 

8. Utilization and disposal of Federal excess and surplus property 
(real and personal), including negotiated disposals. Consideration of 
legislation to improve and broaden the Federal programs for donating 
surplus property and transferring excess property. 

9. Extent of compliance with computer procurement and manage- 
ment provisions of section 111 of the Federal Property Act, which 
provides for the economic and efficient purchase, lease, maintenance, 
operation, and utilization of automatic data processing equipment by 
Federal agencies. 

commerce, consumer, and monetary affairs subcommittee 

House of Representatives, 
Washington, D.C., March 11, 1975. 
Hon. Jack Brooks, 

-Chairman, Government Operations Committee, Rayburn House Office 
Building, Washington, D.C. 
Dear Mr. Chairman : This is in response to your letter of Febru- 
ary 27, 1975, requesting a report from the Commerce, Consumer, and 



399 



45 

Monetary Affairs Subcommittee on its oversight agenda for the 94th 
Congress. 

The oversight agenda outlined below establishes areas of potential 
Subcommittee investigation that I regard as most critical. It is, of 
course, axiomatic that the interests of subcommittee members or future 
events might dictate a departure from this agenda. 

I. BANKING 

(A) Responsibility for the regulation and supervision of federally 
chartered or federally insured banks, savings and loan associations and 
other financial institutions has been entrusted to four major agencies: 
the Federal Reserve Board, the Comptroller of the Currency, the 
Federal Home Loan Bank Board and the Federal Deposit Insurance 
Corporation. The alarming increase in the numbers of bank failures, 
the increase in the numbers of defaulted and foreclosed loans and alle- 
gations of unfair and deceptive banking and other financial institution 
practices suggest the need tor a thorough investigation of the efficiency 
-and economy of the Federal banking regulatory apparatus. Certain 
critics have suggested, for example, that regulatory responsibility is 
excessively fragmented and that bank examination procedures are 
inadequate; 

(B) Of all the Federal regulatory agencies, the financial regulatory 
•entities seem to operate in a way that fosters a minimum amount of 
public accountability and a maximum amount of secrecy. An over- 
sight investigation into the efficiency and accessibility of Federal bank- 
ing agency procedures and practices would seem to be indicated. Of 
particular concern to the Subcommittee is whether the personnel rela- 
tionships between the Federal regulatory entities and the institutions 
they regulate raise conflict-of-interest questions; and consumer infor- 
mation practices and procedures of the Federal banking agencies are 
adequate. 

II. CONSUMER PROTECTION AND INDUSTRIAL CONCENTRATION 

The Federal consumer protection apparatus, particularly the Fed- 
eral Trade Commission's authority to deal with unfair and deceptive 
acts and practices in commerce, determines, to a large extent, whether 
the free enterprise system will operate in a manner that is not only 
efficient but just. Similarly, the success of the free enterprise system 
depends in large measure on the degree to which Federal programs, 
such as those of the Commission's Bureau of Competition, foster 
maximum competition in key industrial areas. Recently, serious ques- 
tions have been raised about the efficiency of FTC rules and procedures 
leading to anti-trust actions and the promulgation of trade regulation 
rules. If Commission rules and procedures impede consumer protection 
and the resolution of possible anti-competitive practices, then it is 
incumbent on the Subcommittee to examine apparent deficiencies. 
Moreover, questions have also been raised regarding the efficiency and 
economy of FTC operations relative to assuring the truthfulness and 
integrity of energy and other advertising. 

III. FEDERAL GOVERNMENT^ ECONOMIC DECISIONMAKING PROCESS 

Fiscal, monetary and regulatory decisions and policies by the Federal 
Government determine, to a significant extent, the success or failure of 



400 



46 

our economic system. Nevertheless, there is enormous proliferation andl 
fragmentation within the economic decision-making processes of gov- 
ernment. Serious questions have been raised regarding the ability of 
our far-flung governmental economic apparatus to meet and be re- 
sponsive to the challenges of our times. While numerous congressional, 
executive branch and other outside investigations have focused on the 
prudence and efficacy of various substantive economic policies, no over- 
sight has been conducted into the organizational apparatus that ulti- 
mately formulates economic policy for the United States. Such an 
investigation is clearly indicated. 

IV. INTERNATIONAL TRADE 

Our Nation's current difficulties with inflation, recession and short- 
ages in key commodity areas and the increased political and economic 
importance of international trade, all point up the need for an over- 
sight review of the efficiency and economy of our international trade 
policies, practices and procedures. The failure of our Federal interna- 
tional trade organizational apparatus to anticipate the effect on Ameri- 
can consumers of unbridled shipments abroad of wheat and key feed 
grains ; issues involving trade in strategic materials ; and allegations 
of U.S. governmental complicity in boycotts and blacklisting based on 
religious heritage, all point up the need for appropriate congressional 
oversight in this area. 

V. SECURITIES AND EXCHANGE COMMISSION/CORPORATE RESPONSIBILITY 

Much attention has been devoted by committees of the Congress and 
industry trade groups to the fair and efficient functioning of our se- 
curities and financial market system. As a result of the efforts by the 
SBC, the stock exchanges and other interested groups, the investing 
public has been provided with much information and malpractices 
have been minimized. But the Securities and Exchange Commission 
has interpreted its legal mandate in a way that tends to exclude the 
issue of corporate citizenship and corporate responsibility. In an age 
when many corporations exercise great influences over the daily lives 
of our citizens and frequently operate as miniature-governments, it is 
important to examine the SEC's practices and procedures for assuring 
that corporate decision-making is made in the broad public interest 
and with appropriate public in-put. 

VI. INTERNAL REVENUE SERVICE TAX MATTER8 

Several issues involving IRS operations seem to require the over- 
sight attention of the Subcommittee : The first has to do with the uni- 
formity of IRS procedures and standards for the informal settle- 
ments of tax claims by and against the Government. The second 
concerns the efficiency or IRS procedures designed to force compliance 
with its prohibition against the deductibility of business expenses; 
associated with influencing congressional or other public policies. The- 
third involves the efficiency of IRS services and assistance to tax 
payers for the reporting and filing of income taxes. 
Sincerely, 

Benjamin S. Rosenthal, Chairman. 



401 



47 

MANPOWER AND HOUSING SUBCOMMITTEE 

House of Representatives, 
Washington, B.C., March, 6, 1975. 
Hon. Jack Brooks, . M ._ 

Chairman, Committee on Government Operations, 2157 Rayburn 
House Office Building, Washington, D.C. 
Dear Mr. Chairman : In accordance with your request of Febru- 
ary 24th, I am enclosing a list of proposed investigations by the Man- 
power and Housing Subcommittee. Two investigations are already 
under way at the staff level. 

The priority that will be assigned to these subject areas will be 
determined at'a Subcommittee meeting to take place later this month. 
I would add the usual caveat that modifications to any plan will 
probably be instituted as a result of future developments and infor- 
mation that comes to our attention. 
Sincerely yours, 

Floyd V. Hicks, Chairman. 

Jurisdiction 

Department 
Department of Housing and Urban Development. 
Department of Labor. 

Other agencies 
Veterans' Administration 
•Civil Service Commission 

Special Action Office for Drug Abuse Prevention 
ACTION 

Federal Mediation and Conciliation Service 
National Labor Relations Board 
National Mediation Board 
Office of Economic Opportunity 
Railroad Retirement Board 
Equal Employment Opportunity Commission 
Occupational Safety and Health Review Commission 

Proposed areas of oversight 

1. Housing: Mortgage Industry's adherence to HUD guidelines 
designed to preclude unnecessary foreclosure on mortgages held by 
low-income home owners. (Already under way.) 

2. The effectiveness of HUD policies and programs for the acquisi- 
tion, retention and disposition of properties owned by the department. 

3. The interrelationship between serious disability claims and safety 
standards for Federal workers in areas of high risk employment. (Al- 
ready under way.) 

4. Effectiveness of Federally funded State unemployment offices 
and employment services. 

5. Labor Department use and audit of past manpower training 
contracts. 

6. The effectiveness of Government agencies' determination of per- 
sonnel requirements and the use of non-career employees for tem- 
porary requirements. 



402 



48 

7. The effectiveness of Federal programs to identify and rehabilitate 
alcoholic workers. (Follow-up investigation.) 

8. Methods used to disseminate effective programs to assist drug 
abusers. 

9. Implementation of recommendations on the Committee's report 
on "Impact of Heroin Addiction on the Criminal Justice System." 

Additional areas are under consideration as a result of Member in- 
quiries posed at the first Subcommittee meeting. Based on past Con- 
gresses, it is likely that several areas for investigation will develop 
over the next few months. 

government information and individual rights subcommittee 

House of Bepresentatives, 
Washington, D.C., March 5, 1975. 
Hon. Jack Brooks. 

Chairman, Committee on Government Operations, Room 2157 Ray- 
burn House Office Building, Washington, D.C. 
Dear Mr. Chairman : The attached is in reply to your letter of 
February 24, 1975, requesting a concise listing of contemplated over- 
sight activities during the 94th Congress by this Subcommittee pur- 
suant to the provisions of House Rule X, clause (2) (c). 

A work agenda was discussed and approved at our Subcommittee's 
organizational meeting on February 19, 1975, which included both 
oversight and legislative hearings for the next several months. Other 
oversight work for the remainder of the 94th Congress is projected, 
on the basis of informal discussions among our Members. 
With best wishes, 
Sincerely, 

Bella S. Abzug, Chairwoman. 

Jurisdiction 

The Subcommittee maintains oversight and legislative jurisdiction 
over the activities and operations of all Federal agencies with respect 
to the Freedom of Information Act (5 U.S.C. 552), the Privacy Act 
of 1974 (5 U.S.C. 552a), and the Federal Advisory Committee Act 
(5 U.S.C. App. 1) 86 Stat. 770. 

It also maintains oversight jurisdiction over the activities and opera- 
tions of the U.S. Information Agency, the National Archives and 
Kecords Service, the Government Printing Office, the Federal Com- 
munications Commission, the U.S. Postal Service, the Department of 
Justice, the U.S. Courts, the Civil Rights Commission, the Smith- 
sonian Institution, the National Foundation on the Arts and the Hu- 
manities, and the American Revolution Bicentennial Administration. 
Oversight targets 

1. Monitoring and oversight activities involving Federal agencies' 
implementation of the 1974 Freedom of Information Act amendments 
(P. L. 93-502). 

2. Monitoring and oversight activities involving Federal agencies' 
implementation of the Privacy Act of 1974 (P. L. 93-579). 

3. Completion of work leading to action by Subcommittee and fulfc 
Committee on draft reports on "Telephone Monitoring Practices of 



403 



49 



Federal Agencies" and "Federal Agencies' Use of Polygraphs and 
Similar Devices." (Based on 1974 investigations and hearings by the 
Subcommittee.) 

4. Preliminary 1-day oversight hearings with new agencies added to 
the Subcommittee's jurisdiction to acquaint Members with the broad 
aspects of each agency's programs, operations, personnel, budgetary 
requirements, and similar general information. These hearings will 
include testimony from : 

U.S. Information Agency 

Department of Justice 

U.S. Postal Service 

Federal Communications Commission 

National Archives and Records Service 

U.S. Civil Rights Commission 

National Foundation on the Arts and Humanities 

Smithsonian Institution 

American Revolution Bicentennial Administration 

U.S. Government Printing Office 

5. Review of Federal agencies' collection and dissemination of sci- 
entific and technical information, overlapping and duplication of such 
efforts and their economy and efficiency'. 

6. Review of the operations of public information offices in key se- 
lected Federal agencies, including extent of "contracting out" of agency 
"public relations" activities. 

7. Review of the scope of planned activities and programs of the 
American Revolution Bicentennial Administration, their economy and 
efficiency and impact on the Bicentennial Year festivities. 

8. Review of the activities of the National Archives and Records 
Service, with particular reference to their work involving declassifica- 
tion of agency documents, Presidential libraries administration, and 
the general economy and efficiency of NARS operations. 

9. Review of the economy and efficiency of operations of the Federal 
Communications Commission and their various statutory responsibili- 
ties in rate-making, broadcasting, and other fields. 

10. Review of the economy and efficiency of operations of the U.S. 
Information Agency, with reference to their status as an independent 
arm of the State Department and recommendations in the forthcoming 
report of the Stanton Commission. 

11. Various other in-depth oversight hearings on those agencies 
listed in paragraph 4 above, depending on information developed dur- 
ing the preliminary orientation hearings, GAO reports, and other 
future developments that may make some of them high priority areas 
for oversight activity. 



404 
COMMITTEE ON HOUSE ADMINISTRATION 



House op Representatives, 
committee on house administration, 

Washington, D.C., March 4-, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, U.S. House of Rep- 
resentatives, Washington, D.C. 

Dear Mr. Chairman : Thank you for your letter of February 13, 
1975 concerning a report from the Committee on House Administra- 
tion on oversight activities of this Committee, pursuant to Clause 2(c) 
of Rule X of the Rules of the House of Representatives. 

Because this overview on oversight responsibilities is required at the 
beginning of the Congress, it should be understood that the Committee 
may weU expand or contract its oversight activities as the need arises. 
It- is now contemplated Committee oversight activities will be orga- 
nized along subcommittee jurisdictions rather than appointing a spe- 
cial oversight committee. The Committee on House Administration 
has nine Standing Subcommittees and two Ad Hoc Subcommittees 
and each will be responsible for oversight in its areas of jurisdiction. 

Oversight activities contemplated by Subcommittees are as follows : 

The Subcommittee on Accounts carries out the oversight responsi- 
bility specified in 2 USC 91 and 95, to ensure the performance of du- 
ties by, and to review the remuneration of, officers and employees of 
the House, and to oversee payments from the contingent fund. 

The Subcommittee on Elections carries out the Committee's over- 
sight responsibilities on matters relating to the election of the Presi- 
dent, Vice President, and Members of Congress; corrupt practices, 
contested elections; credentials and qualification; and Federal elec- 
tions generally. 

The Subcommittee on Library and Memorials has oversight respon- 
sibilities in matters relating to the Library of Congress and the House 
Library, statuary and picture^ acceptance and purchase of works of 
art for the Capitol ; the Botanic Gardens ; management of the Library 
of Congress; purchase of books and manuscripts; erection of monu- 
ments to the memory of individuals and matters relating to the Smith- 
sonian Institution. 

The Subcommittee on Printing has oversight responsibilities deal- 
ing with the printing of House documents ; the Congressional Record ; 
bills concerning the Government Printing Office, the Depository Li- 
brary Program and the use of recycled paper in government docu- 
ments. In addition, the subcommittee will have oversight responsi- 
bilities over the implementation of the Presidential Recordings and 
Materials Preservation Act, monitoring the work of the National 

(50) 



405 



51 



Study Commission on Records and Documents of Federal Officials and 
considering legislation regarding the disposition of documents of Fed- 
eral officials. 

The Subcommittee on Electrical and Mechanical Office Equipment 
has responsibility for evaluating and allocating all office equipment 
used by the House of Representatives as well as procurement 
programs. 

The Subcommittee on Personnel and Police has oversight responsi- 
bilities of the classification and evaluation program for the House of 
Representatives, the Sergeant at Arms, the Doorkeeper, and the Post- 
master. The Subcommittee has further oversight responsibilities with 
regard to the U.S. Capitol Police Board in administering the 1,040 
member force of the U.S. Capitol Police. It will also review the secu- 
rity system for the Capitol complex authorized by House Concurrent 
Resolution 550, 92nd Congress. 

The Subcommittee on Contracts has oversight activities over the 
provisions of section 303 of the Legislative Reorganization Act of 
1970, which pertains to the authorization of standing committees of 
the House to obtain the temporary services of individual consultants 
or organizations to make studies or advise the committee with respect 
to any matter within its jurisdiction. 

The Subcommittee on Parking is charged with oversight responsi- 
bilities regarding vehicle parking facilities and the regulating of park- 
ing spaces for Members and House employees. 

The Subcommittee on Paper Conservation has oversight responsi- 
bility with regard to the distribution and the accumulation of printed 
material in the House of Representatives. 

The Ad Hoc Subcommittee on Computers is charged with oversight 
responsibilities with regard to the House Information Systems and use 
of automated data processing equipment. 

The Ad Hoc Subcommittee on the Restaurants has oversight re- 
sponsibilities with regard to the overall food service facilities of the 
House. 

Doug Frost, Staff Director, and Paula Peak, Deputy Staff Director, 
have been designated as representatives of this Committee to confer 
with Government Operations Committee staff. 

If I can be of any further assistance, please don't hesitate to call 
on me. 

With kind personal regards, I am 
Very sincerely yours, 

Wayne L. Hays, Chairman. 



406 
COMMITTEE ON INTERIOR AND INSULAR AFFAIRS 



U.S. House or Representatives, 
Committee on Interior and Insular Affairs, 

Washington, B.C., February 28, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations^ 2157 Raybum 
House Office Building, Washington, B.C. 

Dear Mr. Chairman : This is in response to your request to provide 
information on oversight activities contemplated by this Committee 
during the 94th Congress. 

In carrying out both the general oversight responsibilities under 
Rule X, Clause 2(b), and the special oversight functions under Rule X, 
Clause 2(e), of the Rules of the House, oversight as well as legislative 
jurisdiction is assigned under our Committee Rules to seven sub- 
committees in their respective areas. However, the Full Committee 
may from time to time initiate oversight inquiries when they appear 
to involve the jurisdiction of several subcommittees. In addition, the 
Committee will designate staff representatives to assist the subcom- 
mittees in fulfilling their oversight role and to work with the Com- 
mittee on Government Operations. 

Rule X, Clause (1), of the Rules of the House sets forth the juris- 
diction of this Committee. Rule 16 of the Rules of the Committee on 
Interior and Insular Affairs further assigns these area6 of general 
Committee jurisdiction to the respective subcommittees. A copy of the 
Committee Rules is enclosed,* and you will note on page 9 that the 
special oversight function with respect to non-military nuclear energy 
and research and development, including the disposal of nuclear waste, 
is given to the Subcommittee on Energy and the Environment; and 
that the special oversight function with respect to all programs affect- 
ing Indians goes to the Subcommittee on Indian Affairs. 

Exercise of the oversight responsibilities and functions of this Com- 
mittee were initiated this month by oversight inquiries into the Alaska 
pipeline construction by the Subcommittee on Public Lands and these 
are expected to continue. On March 4, oversight hearings will be held 
by the Subcommittee on Mines and Mining on Bureau of Mines and 
U.S. Geological Survey programs. On March 5 an orientation meeting 
with the Secretary of the Interior has been scheduled by the Full 
Committee. The Full Committee sessions will be followed by hearings 
March 6, 7, 13, and 14 on national outdoor recreation programs and 
policies to be held by the Subcommittee on National Parks and Recrea- 

•Document available In flies of the Committee on Government Operations. 
(52) 



407 



53 

tion. Later, on March 14, an orientation meeting with the Bureau of 
Reclamation personnel has been scheduled by the Subcommittee on 
"Water and Power Resources. 

Although our detailed schedule is not yet complete for the remainder 
•of this Congress, we do contemplate in addition to those areas to which 
I referred oversight activities by the Subcommittees as follows : 

National Parks and Recreation. — The administration of the National 
Trails Systems Act, the Pennsylvania Avenue Development Plan for 
the nation's Bicentennial celebration, and concessions in the National 
Park System. 

Water and Power Resources. — The implementation of P.L. 84-984 
and P.L. 84-100, having to do with small project loans and distribu- 
tion system loans; the Interior Department's research program; and 
the status of feasibility studies that Congress has authorized in recent 
years. 

Energy and the Environment. — Nonmilitary nuclear energy, re- 
search and development, and the disposal of nuclear waste. 

Territorial and Insular Affairs. — Marianas status negotiations and 
examination of non-essential elements of the organic acts for Guam 
and the Virgin Islands. 

Mines and Mining. — Strip mining regulations on Federal lands as 
well as other oversight functions being carried out as adjuncts to hear- 
ings on major legislation before the Subcommittee. 

Indian Affairs. — The administration of the leasing of Indian trust 
lands, and tribal governmental problems in general. 

Public Lands. — Oil and gas production from public lands (deter- 
mine extent of "shut-in'" capacity) ; Bureau of Land Management 
( BLM) and Forest Service mining regulations as they apply to surface 
protection for public lands ; continuing review of practices, regulations, 
and procedures of BLM and related Forest Service activities; and 
review of forest and range land renewable resources practices. 

In summary, as particularly as may be determined at the present 
time, the Committee on Interior and Insular Atfairs during the 94th 
Congress may probe into any of its areas of jurisdiction in addition to 
those already specifically mentioned; and the Committee may defer 
reviewing some of the jurisdictional areas now being considered for 
oversight activities. In any event, we look forward to cooperating with 
the Committee on Government Operations and will keep you advised 
as our schedule is revised from time to time. 
Sincerely yours, 

James A. Haley, Chairman. 



408 



COMMITTEE ON INTERSTATE AND FOREIGN 
COMMERCE 



House of Representatives, 
Committee ox Interstate and Foreign Commerce, 

Washington. B.C., February 28, 197o. 
Hon. Jack Brooks. 

Chairman, Committee on Government Operation*, 
House of Representative*, 
Washington, B.C. 

Dear Mr. Chairman : As required by Rule X 2(c) of the Rules of 
the House of Representatives, appropriate representatives of the 
Committee on Interstate and Foreign Commerce have met with rep- 
resentatives of the Committee on Government Operations to discuss 
the oversight plans of this committee during this Congress. 

The Committee will be organized during this Congress into six sub- 
committees, with jurisdiction as follows : 
Subcommittee on Oversight and Investigations 

Responsibility for oversight of agencies, departments and all pro- 
grams within the jurisdiction of the full committee and to conduct 
such investigations within such jurisdiction. (Note: This subcommit- 
tee would not have legislative jurisdiction. The committees legislative 
subcommittees would have parallel responsibility to conduct oversight 
in their assigned areas. Rule X(2) (b) of House Rules.) 

Subcommittee on Energy and Power 

Energy matters within the jurisdiction of the full committee, and 
jurisdiction over all petroleum, natural gas, and electrical power 



Subcommittee on Communications 

Jurisdiction over interstate and foreign communications, including 
all communications by satellite, broadcast, radio, common carriers, 
interstate communication bv wire and such jurisdiction over commu- 
nications and media as in tfxe jurisdiction of the full committee. 
Subcommittee on Health and the Environmen t 

Responsibility for public health and quarantine; hospital construc- 
tion; mental health and research; biomedical programs and health 
protection in general, including medicaid and national health insur- 
ance ; foods and drugs ; and drug abuse. 

Jurisdiction over the Clean A.ir Act, the State Drinking Water Act, 
the Radiation for Health and Safety Act and environmental protec- 
tion in general. 

(54) 



409 



55 

Si'txo/nriutter on CvuMtmeV Protection and Finance 

Jurisdiction over consumer protection in general, including toxic 
substances, product safety, motor vehicle safety, securities and ex- 
changes (the SEC), the regulation of trade (the FTC 1 ), and insurance. 
Subcommittee <>,> Transportation and Commerce 

Jurisdiction over the regulation of travel and tourism, interstate 
and foreign commerce generally, and all matters related to inland 
waterways, railroads, railroad retirement, railway labor, solid waste 
disposal ami noise pollution control. 

OVERSIGHT ACTIVITIES, :'•!!) CONGRESS 

In all programs over which this committee has jurisdiction where 
it is appropriate to do so. the committee reports legislation to the 
House which is limited in duration, generally to three fiscal years. 
This means that at least once every three years, a majority of the pro- 
grams which this committee is responsible for are reviewed, during 
the process of committee consideration of legislation extending the 
duration of the programs. A listing of these programs which were 
within the committees jurisdiction during the 03d Congress is set 
forth on pages 110-238 of the committee's activity report covering the 
93d Congress (H. Kept. 93-1668). 

In addition, each of the legislative subcommittees of the full com- 
mittee held legislative oversight hearings during the 93d Congress on 
one or more matters within the subcommittee's jurisdiction. These 
hearings covered the Florida power interruption, the Civil Aeronau- 
tics Board. Independent truckers and the energy crisis, the National 
Highway Traffic Safety, the Motor Vehicle Information and Cost 
Savings Act. the Consumer Product Safety Commission, HEW health 
programs, regional medical programs. Community Mental Health 
Centers. Public Health Service Hospitals, HEW Reorganization, 
Clean Air Act. Drug Abuse. Motor Vehicle standards and fuel econ- 
omy, national health care and hospital rate regulation and Phase IV 
cost controls for health services. 

In addition to the foregoing, the Special Subcommittee on Investiga- 
tions conducted numerous oversight activities, summarized on pages 
61 through 70 of the committee's activity report (II. Rept. 93-1668), 
and held hearings on the suhjects of freight car shortages, SEC trans- 
fer of ITT files to the Justice Department, ex parte contacts with the 
SEC, the liquefied natural gas tank disaster on Staten Island, FTC 
practices and procedures, FA A air safety oversight. FAA-transporta- 
tion of hazardous materials. FAA-reliability of "drilled" turbine fan 
blades on CF-6 Kiiirine. FAA-Ground Proximity Warning System. 
FPC re GAO report on gas rates, and an inquiry into CAB rate in- 
creases. 

I anticipate that similar activities will be engaged in by the sub- 
committee during the coming Congress. 

ANTICIPATED OVERSIGIIT ACTIVITIES, D4TTI CONGRESS 

During the 94th Congress. I anticipate that the Committee, either 
as a whole, or through one or more of its subcommittees, will engage 



410 



56 



in oversight activities, including hearings, involving the Federal Com- 
munications Commission, the Office of Telecommunications Policy, the 
communications satellite program, energy, health care costs, HEW 
health programs, both generally and specifically, the administration 
of laws by the independent regulatory agencies over which the com- 
mittee has jurisdiction, problems involving the railroads, specifically 
the problems facing the Penn Central and other northeast railroads, 
responsibility for public health and health protection in general, in- 
cluding hospital construction; health manpower; nurse training and 
nursing home construction ; mental health ; biomedical research ; drug 
abuse; health services programs; the Public Health Service; Indian 
health; laws affecting foods, drugs, cosmetics and medical devices; 
Medicaid; and national health insurance proposals funded by reve- 
nues not derived from payroll deductions, among others. 

It is, of course, too early in this session of the Congress to determine 
what the full scope of oversight activities by the committee and its 
subcommittees will be ; however, based upon the past, I would antic- 
ipate that there will be some degree of oversight activities covering 
most of the programs within the committee's jurisdiction, and over- 
sight hearings covering a substantial number of them. 
Sincerely yours, 

Hakuey O. Staggers, 

Chairman. 



411 
COMMITTEE ON THE JUDICIARY 



House of Representatives, 
Committee on the Judiciary, 
Washington, D.C., March 3, 1975. 
Hon. Jack Brooks, 

Chairman, Committer on Government Operations, House of Repre- 
sentatives, Washington, D.C. 
Dear Jack: In accordance with Clause 2(c) of Rule X of the 
Rules of the House and staff discussions, I enclose the oversight pro- 
gram of the Committee on the Judiciary for the 94th Congress. Each 
subcommittee will exercise oversight responsibilities within its area 
of jurisdiction. 

The enclosed oversight program sets forth general oversight juris- 
diction of each subcommittee as well as specific oversight projects 
proposed to be undertaken by each subcommittee. It is possible that 
additional oversight projects within the jurisdiction of the Committee 
may be undertaken, and also that some projects may be deferred or not 
be considered during the 94th Congress. 
Kind regards. 
Sincerely, 

Peter W. Rodino, Jr., Chairman. 
Enclosure. 

subcommittee on immigration, citizenship, and international law 

The Subcommittee is responsible for reviewing on a continuing basis 
the administration of the Immigration and Nationality Act by the 
Departments of State, Justice, and Labor. Several hearings were con- 
ducted during the 92nd and 93rd Congresses to review the operation 
of this Act including pertinent regulations, operating instructions, 
and interpretations of that Act which have been promulgated by the 
Executive Branch. The Subcommittee expects to continue to closelv 
monitor the implementation of this basic immigration law and will 
attempt to identify problems which have developed in the adminis- 
tration of this law. 

The Subcommittee also intends to review the operation of the Sub- 
merged Lands Act and the administration of the Outer Continental 
Shelf Lands Act which it originally processed in 1953. Preliminary 
hearings were conducted in the second session of the 93rd Congress 
and additional hearings are planned in order to determine whether 
amendments to this basic legislation are required. The administra- 
tion of this Act has become particularly important in view of the 
Administration's plans to accelerate the leasing of oil and gas on 
the Outer Continental Shelf. 

(57) 



412 



58 

The Subcommittee plans to conduct a comprehensive investigation 
of the foreign student issue. This includes the issuance of student 
visas to aliens, the role of the educational institutions in the visa is- 
suance process, the efforts of the Immigration and Naturalization 
Service to monitor the foreign student program, the procedures for 
granting off -campus and summer work permission to foreign students. 
the approval and withdrawal of approval for educational institutions 
receiving foreign students, and a variety of other problems closely 
associated with the foreign student program. 

The Subcommittee will closely scrutinize the various refugee prob- 
lems which have developed around the world and will conduct hearings 
to review the operation of the Migration and Refugee Assistance Act 
of 1962. In addition, the Subcommittee will study the relationship 
between the 1962 law and the emergent refugee problems which have 
developed in various countries (i.e., Soviet Jews, Chilean refugees, 
Haitian refugees, refugees from mainland China). In this regard, the 
Subcommittee intends to review procedures for the admission of 
refugees to the United States as well as programs which have been 
established for the movement and resettlement of various categories 
of refugees. 

Another topic which will be pursued by the Subcommittee in this 
Congress is the operation of the labor certification program prescribed 
in Section 212(a) (14) of the Immigration and Nationality Act. The 
Subcommittee is expected to hold several hearings to consider the 
utility of the labor certification requirement as well as its implementa- 
tion by the Departments of Justice and Labor. 

Finally, the Subcommittee will study the provisions of the Immi- 
gration and Nationality Act relating to the exclusion and deportation 
of those who will or have become public charges. 

The aforementioned oversight projects are those which the Sub- 
committee intends to carry out at the present time. 

SUBCOMMITTEE ON ADMINISTRATIVE LAW AND- GOVERNMENTAL RELATIONS 

The oversight jurisdiction of the Subcommittee over the Depart- 
ment of Justice relates primarily to the Civil Division, with emphasis 
on those actvities involving torts, claims by and against the Govern- 
ment, and contracts. Justice Department oversight also includes lands 
division activity, with reference to the recently-enacted law providing 
jurisdiction in the Federal courts to try land titles involving the 
United States. The claims jurisdiction of the Subcommittee requires 
oversight over the administration of the claims statutes by executive 
departments and agencies, including the military departments and 
their operations both in this country and abroad. Also included in its 
responsibility is oversight of the Secret Service relating to protection 
of the President and other persons. Incident to its iurisdiction over 
legislation relating to the termination of National Emergencies, the 
Subcommittee has oversight over the administration of operation of 
the various agencies of Government under emergency statutes, with 
particular reference to the Department of Defense and the Treasury 
Department. Under its jurisdiction over administrative law, the Sub- 
committee will be required to exercise jurisdiction over those depart- 



413 



59 

ments and agencies operating under the Administrative, Procedure 
Act and other functions falling within the area of administrative kiw. 
The Subcommittee contemplates a long-term examination of the 
process of adjudication and decision-making in Government in the 
area of administrative law and practice. This will include an exami- 
nation of the current application of the Administrative Procedure 
Act but will extend beyond that to a basic analysis of adjudication 
and settlement. Under multiple laws, the agencies and departments of 
Government settle claims arising from Government activity. This 
too will be a continuing study with the immediate examination relat- 
ing to the Tort Claims Act. The emergency powers operations of 
Government are of immediate concern to the subcommittee and will 
be the subject of examination in the near future. 

SUBCOMMITTEE ON COURTS, CIVIL LIBERTIES, AND THE ADMINISTRATION" 
OF JUSTICE 

Pursuant to jurisdiction assigned by Rule VI of the Committee's 
Rules of Procedure, the Subcommittee plans to exercise general over- 
sight in the 94th Congress with regard to the Federal Bureau of Pris- 
ons, the Patent Office, the Copyright Office, the Federal Court System, 
the Presidential Clemency Board, and Government agencies conduct- 
ing surveillance of American citizens. Specific oversight projects now 
being planned or already initiated include Bureau of Prisons Con- 
struction Plans and Behavior Modification Program; Federal Pris- 
on Industries; the Presidential Clemency Board; electronic surveil- 
lance by Federal agencies and the Justice Department pilot project 
on pretrial diversion. 

Specific legislative measures to be considered will include a variety 
of proposed improvements in the corrections systems, a copyright law 
revision, and a variety of proposals to improve the Federal Court 
System. 

It must, of course, be understood that the foregoing is a highly ten- 
tative estimate, subject to change with changes in the legislative scene. 
This may include removal of subjects from consideration as well as 
the addition of further areas of study. 

SUBCOMMITTEE OX CIVIL AND CONSTITUTIONAL RIGHTS 

The Subcommittee on Civil and Constitutional Rights has been 
assigned oversight responsibilities for the Federal Bureau of Inves- 
tigation, the Civil Rights Division of the Department of Justice and 
the Community Relations Service. 

The Subcommittee has undertaken and is presently pursuing the 
following oversight projects : 

1. The Operations of the FBI 

Hearings, staff and GAO studies are underway to determine the 
economy, efficiency, and effectiveness of the FBI.' The areas of con- 
centration at the moment are: 

(a) the coimterintelligenee programs; 

(b) information gathering and record keeping activities; 
(r) statistical reporting systems and procedures; 

(d) the authority underlying all FBI activities. 



414 

60 

£. fivil Rights Oversight 

(a) Staff studies and hearings are underway relative to the 
enforcement of the civil rights provisions of the act commonly 
known as General Revenue Sharing. 

(b) Oversight is continuing in the areas of Federal employment 
problems of Spanish-speaking Americans and particularly in 
the area of enforcement of the Voting Rights Act by the Civil 
Rights Division of the Department of Justice. 

(c) Oversight is also continuing in the area of equal housing 
opportunities and in the enforcement of Title VI of the Civfl 
Rights Act of 1964 in programs receiving medicare and medicaid 
funds. 

SUBCOMMITTEE ON MONOPOLIES AND COMMERCIAL LAW 

The Monopolies and Commercial Law Subcommittee has general 
oversight responsibilities in the field of enforcement of the antitrust 
laws. This includes the activities of the antitrust division of the De- 
partment of Justice and the Federal Trade Commission, as well as a 
variety of other agencies responsible for the administration of pro- 
grams and policies within the framework of antitrust enforcement or 
embodying exemptions from the antitrust laws. 

The Subcommittee envisions major hearings this year on the en- 
forcement of the antitrust laws in the general field of development and 
distribution of energy resources. These hearings will focus specifically 
on the problems posed by the emergence of so-called "energy conglom- 
erates". The Subcommittee further anticipates review of the applica- 
tion of the new criminal provisions of the antitrust laws enacted by the 
previous Congress, a study of the need for expansion of the power of 
the antitrust division under the Antitrust Civil Process Act, and a 
general review of the interplay between private and governmental (in- 
-cluding both state and federal) enforcement of the antitrust laws. 

SUBCOMMITTEE ON CRIME 

The Subcommittee on Crime has oversight jurisdiction over the law 
Enforcement Assistance Administration and the Drug Enforcement 
Administration. The Subcommittee will exercise its responsibility with 
respect to LEAA beginning in midyear 1975. The staff is presently in- 
vestigating the following areas for LEAA oversight hearings : LEAA 
and its response to the problems of state courts, law enforcement and 
corrections ; effectiveness of block and discretionary grant programs ; 
long-term impact of Law Enforcement Assistance grants ; and evaluat- 
ing the results of LEAA programs. 

With respect to the Drug Enforcement Administration, the Sub- 
committee staff is reviewing the following: areas for oversight hear- 
ings : jurisdictional problems in the Drug Enforcement between DEA 
and Customs : and effectiveness of DEA overseas operations. 

In addition to the above areas, the Subcommittee may consider other 
possible subjects for oversight hearings. The final decision awaits re- 
view by the Subcommittee. 



415 



61 

SUBCOMMITTEE ON CRIMINAL JUSTICE 

The Criminal Justice Subcommittee has been assigned oversight 
responsibilities for the activities of the Justice Department's Criminal 
Division. The Subcommittee has asked Acting Assistant Attorney Gen- 
eral John C. Keeney to prepare a report on the organization and oper- 
ation of the Criminal Division. 

The Subcommittee is meeting on March 12 to review this report. At 
that time, the Subcommittee will decide what areas of Criminal Divi- 
sion activities to look into. There has been interest expressed in review- 
ing the exercise of prosecutorial discretion in conspiracy prosecutions 
and in reviewing the use of the witness immunity provisions of Title 
18. Oversight hearings in both of these areas would be carried out in 
conjunction with the Subcommittee's work on criminal code reform 
legislation. 

The Criminal Justice Subcommittee will have to maintain close liai- 
son with the Subcommittee on Courts, Civil Liberties, and the Admin- 
istration of Justice, which has oversight responsibility for the activi- 
ties of U.S. Attorneys. Prosecutions under many of the statutes that 
the Criminal Division has jurisdiction over are carried out by U.S. 
Attorneys. Thus, the Criminal Justice Subcommittee's oversight activi- 
ties may overlap those of the Subcommittee on Courts, Civil Liberties, 
and the Administration of Justice. 



416 
COMMITTEE ON MERCHANT MARINE AND FISHERIES 



U.S. House of Representatives, 
Committee ox Merchant Marine and Fisheries. 

Washington. D.C. February 28. 197'j. 
Hon. Jack Brooks. 

Chairman, Hovse Government Operations Committee. 2157 Rayhvrn 
House Office Building, Washington, D.C. 

Dear Mr. Chairman : Tliis is in reply to your letter of February 13, 
1975, in which you requested a list of oversight activities contem- 
plated by the Committee on Merchant Marine and Fisheries during 
the present Congress. The information is needed in order to enable 
you to fulfill your responsibilities under Clause 2(c) of Rule X of 
the Rules of the House. 

In adopting its Committee Rules for the 91th Congress, the Com- 
mittee on Merchant Marine and Fisheries has specifically incorporated 
by reference the oversight provisions of the House Rules. In per- 
forming its oversight functions, the Committee has elected to operate 
through its regular subcommittees without the establishment of a 
Subcommittee on Oversight. Each Subcommittee has subsequently 
developed plans for performing oversight in its own area of jurisdic- 
tion. 

I am sure that you will realize that our present oversight plans, 
while generally complete as we see them at this time, will necessarily 
be subject to change with changing circumstances during the course 
of the next two years. Furthermore, changes in priority may cause 
adjustments in future activities. Subject to that caveat, the Commit- 
tee oversight plans are reasonably firm and are listed in the attached 
material, divided generally into areas of Subcommittee activity. I 
should point out in that connection that, in addition to the specific 
listings, two of the Subcommittees perform jreneral agency oversight 
in the consideration of annual authorization bills involving the Mari- 
time Administration and the Coast Guard. 

I trust that the material as presented is consistent with your require- 
ments. 

Sincerely, 

LeoxorK. (Mrs. Johx B.) Sullivan. 

Chairman. 

a. subcommittee ox coast guard and xavigattox 

1. Marine safety generally, including: measures relating: to commer- 
cial vessel construction, loadlinos. navigational rules and personnel 
licenses, and recreational boating construction standards and opera- 
tions. 

(62) 



417 



63 

2. Marine traffic management, and particularly vessel traffic systems 
in U.S. navigable waters. 

3. Pollution control measures, including specifically tank vessel 
construction and operation. 

4. Marine law enforcement generally, including ant i -smuggling, 
offshore fisheries laws and treaties, and ocean dumping, as well as the 
implementation of the Deep Water Port Act of 1974 as it relates to 
safety and anti-pollution measures in particular. 

5. Coast Guard procurement practices. 

B. SUBCOMMITTEE ON FISHERIES AND WILDLIFE CONSERVATION' AND THE 
ENVIRONMENT 

1. Commercial fishing generally, arid the implementation of fish- 
ing management practices. 

2. National wildlife refuge administration. 

3. Implementation of fishing treaties and conventions, including 
agreements on coastal fisheries, high seas salmon fisheries, and shrimp 
fisheries in foreign waters. 

4. Implementation of various laws, including — 

National Environmental Policy Act of 1969 
Marine Mammal Protection Act of 1972 
Endangered Species Act of 1973 
Migratory* Bird Acts 
Fish and Wildlife Coordination Act 
Wetlands Acquisition Act of 1961 
Fishermen's Protection Act of 1967 
and various laws related to conservation programs. 

C. SUBCOMMITTEE ON MERCHANT MARINE 

1. Maritime Administration generally, and the Federal Maritime 
Commission. 

■2. Maritime education and training. 

3. Maritime research and development. 

4. Maintenance of the National Defense Reserve Fleet. 

5. Financial responsibility requirements for foreign vessels operat- 
ing in United States waters. 

6. Implementation of the Merchant Marine Act, 1936, as amended 
relating particularly to construction subsidy funds, negotiated con- 
tract of subsidized ship construction, operating subsidy funds, the loan 
guarantee program, and the war risk insurance program. 

7. Implementation of the Shipping Act, 1916, as amended, relating 
particularly to conference activities, dual rate contract activities the 
licensing and regulation of ocean freight forwarders. 

D. SUBCOMMITTEE ON OCEANOGRAPHY 

1. Marine resource development, including Outer Continental Shelf 
resources and the impact of their exploitation on the marine 
environment. 

2. Oceanographic research generally, including adequate safetv 
measures for oceanographic research vessels and platforms. 



418 



64 

3. National Advisory Committee on Oceans and Atmosphere, and 
the implementation of its recommendations. 

4. Implementation of the Marine Protection. Research, and Sanctu- 
aries Act of 1972, particularly in relation to alternatives for solid 
waste disposal, in ocean areas. 

5. Implementation of the Coastal Zone Management Act of 1972, 
particularly in its relation to activities on the O • -ter Continental Shelf 
and the impact of those activities on coastal states. 

6. Implementation of the Sea Grant College and Program Act 
of 1966. 

E. SUBCOMMITTEE ON THE PANAMA CANAL 

1. Personnel policies of the Panama Canal Company and the Canal 
Zone Government. 

2. Civil operations of the Canal Zone Government. 

3. Tolls, toll structures, and financial management of the Canal. 

4. The relationship of the Panama Canal to world shipping and to 
national security. 

5. Implementation of the Coastal Zone Management Act of 1972. 

6. Labor-management relations in the Canal Zone. 

7. Implications of treaty negotiations on future Canal Zone oper- 
ations. 



419 
COMMITTEE ON POST OFFICE AND CIVIL SERVICE 



U.S. House of Representatives, 
Committee on Post Office and Civil Service, 

Washington, D.C., February 25, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, House of Repre- 
sentatives, Washington, D.C. 
Dear Mr. Chairman : As requested by your letter of February 13, 
1975, there is attached the report of the Post Office and Civil Service 
Committee listing the oversight activities contemplated by the Com- 
mittee during the 94th Congress. 
With kindest regards, I am 
Sincerely yours, 

David N. Henderson, Chairman. 
Enclosure. 

LEGISLATIVE OVERSIGHT ACTIVITIES PROPOSED FOR THE 94TH CONGRESS 

Committee Organization 

Rule 20 of the Rules of the Post Office and Civil Service Commit- 
tee establishes six subcommittees, each with both legislative and over- 
sight responsibilities for the matters under their respective jurisdic- 
tions, as provided under Committee Rule 21. In addition, Rule 20 
authorizes the Chairman, or the Committee by a majority vote, to es- 
tablish special or ad hoc investigative subcommittees and assign to 
them such jurisdiction as the Chairman of the Committee deems neces- 
sary. No special or ad hoc investigative subcommittee has been estab- 
lished at this time. 

Scope of Oversight Responsibilities 

The Committee has continuing oversight responsibilities, under 
House Rule X(o) , over the United States Census, United States Postal 
Service, Postal Rate Commissions^ and the United States Civil Service 
Commission, and the programs administered by the agencies involved. 
In the case of the Civil Service Commission, this includes the Federal 
Civil Service and the competitive Civil Service generally, Federal em- 
ployee pay and fringe benefit matters, including Civil Service retire- 
ment, life and health insurance programs; employee political rights 
(Hatch Act matters) ; and the intergovernmental Personnel Programs. 
(65) 



420 



66 

Orcrxight Program for the 9^th Congress 

The Committee, acting through the six subcommittees, has adopted 
an. oveisight program on the subject matters outlined below : 
Manpower and Ciril Service Subcommittee 

1. Violations and abuses of the Federal Civil Service merit prin- 
ciples, including examination and_recruitment for high level positions: 
relationship between career Civil Service and political appointees: 
review and evaluation of agency controls to insure compliance with 
the merit system: and effectiveness of the controls for determining 
whether particular positions should be exempted from the competitive 
service. 

2. The system of examining and issuance of certificates of eligibility 
for competitive Civil Service positions and the use of the three top 
eligibles for such certificates of eligibility. 

3. The ranking of positions under the Executive Salary Schedule 
(5 U.S.C. 5311-5317). 

-k Personnel and pay systems of the Panama Canal Company and 
their l-elationship to personnel and pay systems for employees of other 
executive departments in the Canal Zone. 

5. The impact of personnel systems and pay policies on employees 
of the scheduled disestablishment of Headquarters Pacific Air Forces 
and U.S. Army Headquarters Pacific. 

G. Civilian manpower utilization studies, including the review sys- 
tem for determining "in house" versus technical support service con- 
tract requirements; the use of statutory personnel ceilings versus 
effective dollar economies; and the proper use of civilians in positions 
occupied in the Department of Defense by military personnel. 
Postal Service Subcommittee 

1. The financial status of the United States Postal Service, including 
proposals for increased appropriations for subsidies ; the use of bond 
revenues for operating expenses; and the allocation of cost between 
the various postal services. 

2. Review of the function and operation of the Board of Governors 
of the Postal Service. 

3. Review of the organization and operation of the United States 
Postal Rate Commission, including the hearing procedures; appoint- 
ment and qualifications of the Commissioners; and the authority of 
the Commission to prescribe the procedure and information to be 
furnished by the Postal Service in a postal rate or classification 
hearing. 

4. A review of the application of the Private Express Statutes. 

5. Review of the standards and methods of delivery of the United 
States mail and the handling of military mail; and review of postal- 
related services, including the sale of envelopes and packing materials. 
Postal Facilities. Mail, and Labor Management Subcommittee 

1. Review of the labor-management issues in the Postal Service. 
including the right to strike, uniformity in labor relations, an arbitra- 
tion board system to settle disputes with postal supervisors, and safety 
programs for postal employees. 

2. Review of the Letter Carrier Route Evaluation Svstem and the 
system for central mark-up of mail. 

3. A review of the postal facilities improvement and mechanization 
program and the real estate acquisition policies. 



421 



4. A review of the mailability of particular mail matter, including 
obscene material, lottery information, and unsolicited drugs. 

... A review of tin- program for the standardization of envelopes, 
and size and weight limits for parcels. 

(i. The impact on the United States mails of the electronic trans- 
mission of information. 

7. Review of the program for acquisition of data processing 
equipment. 
Retirement and Employee Benefits Subcommitfi e 

1. Review of the financial condition of the Civil Service Retirement 
Fund and the procedures of investing the current surplus funds. 

2. A review to compare the Federal Employees' Life Insurance Pro- 
gram benefits and costs with comparable program benefits and costs in 
private industry. 

3. A review of the pay comparability system used for adjusting the 
pay of Federal employees under the statutory salary systems. 

4. A review of certain matters related to the Health Benefits Pro- 
gram, including inequities in benefits and costs arising because of 
certain restrictions in State laws; the advisability of extending the 
program to include coverage by additional insurance companies on the 
basis of competitive bidding; a review of the administration of the 
program by the Civil Service Commission; and the feasiblity of con- 
verting the program to a Federally owned and operated program. 

Employee Political Rights and Intergovernmental Programs Subcom- 
mittee 

1. A review of the political rights of Federal employees under the 
Hatch Act and the administration of that Act by the Civil Service 
Commission. 

2. Review of the administration and the operation of the Intergov- 
ernmental Personnel Program, including the adequacy of the present 
limitations on the grant-in-aid authority; the exchange of Federal 
and State employees; and Equal Employment Opportunity in local 
personnel systems. 

Census and Population Subcommittee 

1. A review of the plans of the Bureau of Census for the 1980 census 
program, including the protection and the confidentiality of the census 
information. 

2. A review of the statistics collection program of the Bureau of 
Census, including the collection of unemployment statistics. 
Future Oversight Review Programs 

The Committee has the policy of reviewing recommendations in- 
volving the agencies under our oversight jurisdiction included in 
General Accounting Office reports. Also, the Committee is continuallv 
interested in current reports issued by the Civil Service Commission 
on the operations of the Civil Service by agencies of the Executive 
Branch and of operations under the retirement and life insurance pro- 
grams. Any one of these matters may result in additional programs 
that would be proper for a legislative review by one of our subcom- 
mittees at am ' ' 



ny time during the period covered by this report. 
Also, the Committee will continue to exercise its legislativ 



e over- 



sight function with regard to :ny matter under the Committee's juris- 
diction involving circumstances justifying a review bv the Committee 
during the 94th Congress. 



422 



COMMITTEE ON PUBLIC WORKS AND 
TRANSPORTATION 



House of Representatives, 
Committee on Public Works and Transportation, 

Washington, D.C., March 3, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, 
House of Representatives^ Washington, D.C. 

Dear Jack : Enclosed is a memorandum of information in response 
to your recent request for the Committee on Public Works and Trans- 
portation's oversight responsibilities and activities. 

I trust this information will be helpful to you and that you will 
contact me if I can supply you with any additional information or be 
of assistance to you in any way. 

With every good wish and warm personal regards, 
Sincerely, 

Robert E. Jones, Chairman. 
Enclosure. 

MEMORANDUM 

Subject: Oversight Activities of the Committee on Public WorKs 
and Transportation, 94th Congress. 

I. Mechanics of Oversight. — The Committee has had, since Sep- 
tember 15, 1959, a fully functional Subcommittee on Investigations 
and Review. This subcommittee is comprised of 22 members and has 
a staff of 19. This subcommittee and the appropriate subcommittee 
with legislative authority cooperatively review and study, on a con- 
tinuing basis, the application, administration, execution, and effec- 
tiveness of those laws, or parts of laws, within the Committee's juris- 
diction. The oversight may be carried out by the Subcommittee on 
Investigations and Review, by the legislative subcommittee or both. 
In similar fashion, the Federal agencies and entities having respon- 
sibilities for the administration and execution of these laws are re- 
viewed in order to determine whether such laws and the programs 
thereunder are being implemented and carried out in accordance with 
the intent of Congress and whether such programs should be con- 
tinued, curtailed, modified or eliminated. 

II. General Outline of Oversight Responsibilities. — The Commit- 
tee has oversight responsibilities in the following general program 
areas: National transportation policy; civil aviation, including Air- 
port Development and the activities of the Civil Aeronautics Board, 
the Federal Aviation Administration, and the National Transporta- 
tion Safety Board ; surface transportation, including highways, mass 

(68) 



423 



ay 



transportation, highway motor carrier and motor vehicle safety, and 
the activities of the Interstate Commerce Commission; development 
of the nation's water resources, including water pollution control, 
deep water ports, the activities of the U.S. Army Corps of Engineers, 
the St. Lawrence Seaway Development Corporation and the Ten- 
nessee Valley Authority ; the Economic Development Administration 
and related economic development programs, including Appalachian 
regional development ; watershed program of the Soil Conservation 
Service; public buildings and grounds; uniform relocation assistance; 
and disaster relief. 

III. Likely Areas of Concentration in the 9^th Congress. — Imple- 
mentation of Federal Water Pollution Control Program, highway 
and air safety, transportation regulatory reform, construction over- 
runs, water resources project evaluation, disaster relief, relocation as- 
sistance program, public mass transportation, and p reconstruction 
delays. 

IV. It should be noted that the above statements are general in 
nature and are intended to be neither all inclusive nor unchangeable. 
It is possible, if not likely, that the course of events during the next 
two years will cause the Committee to apply its oversight capabilities 
in areas not specifically foreseen at this time and to place less em- 
phasis than is now anticipated on areas mentioned herein. 



424 
COMMITTEE ON RULES 



U.S. House of Representatives, 

Committee on Rules, 
Washington, D.C., February 28, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, U.S. House of 
Representatives, Washington, D.C. 

Dear Mr. Chairman: Chairman Madden is happy to submit the 
following information in line with your request for certain plans for 
operation of the House Committee on Rules for 1975. 

Although committee rules have been adopted and the House Rules 
Committee is functioning, the issue of additional staff has been post- 
poned and the Committee has not decided to operate with subcommit- 
tees to date. The Rules Committee is exempt from the requirement to 
form subcommittees. However, the jurisdiction and responsibility of 
the Rules Committee has been broadened considerably by the new 
changes in the House Rules. 

One of the most important new assignments of the House Rules 
Committee is its general oversight responsibilities which it has never 
had before. "Each standing committee . . . shall review and study, on 
a continuing basis, the application, administration, execution, and 
effectiveness of those laws, or parts of laws, the subject matter of which 
is within the jurisdiction of that committee, and the organization and 
operation of the Federal agencies ... In addition, each such commit- 
tee shall review and study any conditions or circumstances which may 
indicate the necessity or desirability of enacting new or additional 
legislation . . ." 

This oversight function clearly includes the Legislative Reorganiza- 
tion Act of 1970 and the Congressional Budget and Impoundment 
Control Act of 1974. which this Committee wrote. It also includes 
oversight of Boiling Resolution. H. Res. 988. 

(a) The new Rules give the House Committee on Rules the respon- 
sibility for emergency waivers of the required reporting date for 
bills and resolutions authorizing new budget authority. This means 
that the Rules Committee should follow the new legislative Budget 
Committee in its progress very closely on a continuous basis in order 
to be in a position of expediting its operations in case of difficulties 
with the other legislative committees, the other body, or in other pos- 
sible situations which may arise to cause the House Budget Commit- 
tee to be inoperative. 

(b) The Joint Committee on Congressional Operations is assigned 
the responsibility of conducting a continuous study of the jurisdiction 
of the various standing committees of the House and must report 

(70) 



425 



71 



periodically its results to the House Committee on Rules. It is in- 
cumbent upon the House Committee on Rules to consider these studies 
and recommendations relative to updating the jurisdictions of the 
various standing committees. Evidently, there is a continuing need 
in this area. For example, H.R. 2633, a bill to increase domestic energy 
supplies and availability, was referred to four different committees 
under the new rules. This will not simplify a hearing before the Rules 
Committee if or when the Armed-Services, Interstate and Foreign 
Commerce, Ways and Means, and the Banking, Currency, and Hous- 
ing Committees testify on this measure. Some legislation has been 
referred to five committees. 

(c) These Rules changes and specific assignments, along with the 
nature tit the overall across-the-board consideration for rules on every 
conceivable subject from twenty legislative committees, seem to put 
the House Rules Committee in a position of monitoring or perform- 
ing a broad oversight function — especially as it relates to the legisla- 
tive budget operations, the Joint Committee on Congressional Opera- 
tions, and perhaps other pressing issues that are disturbing our coun- 
try. This latter responsibility seems to be tied in with the authority of 
the House Rules Committee to discharge other committees on issues 
of national or international consequences which are bogged down 
somewhere along the legislative process. 

The Chairman and the Committee staff are happy to cooperate with 
you in this undertaking and hope that the above information is ap- 
propriate to your request. 

Rest personal regards. 
Sincerely, 

L. C. Battle. Staff Director. 



426 
COMMITTEE ON SCIENCE AND TECHNOLOGY 



U.S. House or Representatives, 
Committee on Science and Technology, 

Washington, D.C., February 28, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, House of Repre- 
sentatives, Washington, D.G. 

Dear Mr. Chairman : In response to your letter of February 13, 1 
am reporting the oversight activities contemplated by the Committee 
on Science and Technology during the 94th Congress. 

Oversight is a responsibility of the Legislative branch which the 
Committee on Science and Technology takes very seriously. A contin- 
uing review of the administration, execution and effectiveness of the 
federal agencies and laws enacted for which this Committee has been 
given responsibility is carried out by the Subcommittees when there 
are well denned areas of jurisdiction. Where there is overlap of juris- 
dictions, oversight is retained for full Committee action. 

General oversight activities of the Committee include a continuing 
review of the National Aeronautics and Space Administration, the 
National Science Foundation, the National Bureau of Standards, the 
non-nuclear portion of the Energy Research and Development Admin- 
istration and the Research and Development programs of the Environ- 
mental Protection Agency. In addition, the Committee Reform 
Amendment of 1974 gave the Committee on Science and Technology 
the responsibility of Special Oversight over all non-military research 
and development. This is a comprehensive responsibility which the 
Committee intends to exercise aggressively and yet wisely. 

Specific oversight reviews and inquiries are planned in the follow- 
ing areas of primary and special oversight jurisdictions. 



International cooperative programs in Space. 

Need for international agreements and conventions to regularize com- 
mercial space activities 
Need to provide body of law consistent with multinational space re- 
lated needs 
NASA Operations including : 

Contracting procedures and approaches 
Institutional Support Contracting 
Facilities utilization and consolidation 
Recruiting, training and retention practices 
(7-. 



427 

73 

NASA Management including: 

The "lead center" management concept for the Space Shuttle 

Planning, organization, direction and control of programs and 
institutions 

Procurement management 
NASA Programs including: 

Space Shuttle, Viking, LandSat, ATS 

Tracking and Data Relay Satellite 

Program planning and control techniques 

Future programs — plans, opportunities and alternative objectives. 

AVIATION 

Agency duplication in Aviation R&D programs and facilities. 

FAA and NASA R&D programs : objectives and content. 

Aviation Safety R&D. 

Advanced Aircraft Systems: Subsonic, Supersonic and Hypersonic. 

Aviation Military Aeronautical R&D Relationships. 

Air Traffic control : Long Term technology needs. 

National Aeronautical R&D Facilities Program. 

Alternative Aviation Fuels and fuel conservation technology. 

Microwave Landing System : Technology issues and status. 

Airport design and development. 



National Science Foundation : 

Research Applied to National Needs : Program users and utility of 
programs. Analysis of grant methodology of basic research 
projects 
Science Education programs and cutback effects 
Environmental data handling systems 

R&D incentive program and its application 
Development of a National Science Policy, including the planning and 
organizational structure for handling Federal R&D as well as science 
advice to the President 
Strengthening the national economy through the utility of Research 

and Development 
Application of Science and Technology to Urban Problems 
Materials Shortages and Materials Research and Development 
Technology Transfer and International Trade 
American-Soviet Technology Transfer 
American-Soviet Science Cooperation 
Science in China 
Federal/local relations in Science and Technology 

ENVIRONMENT 

Adequacy of inter-agency coordination in environmental R&D 
Long term effects on climate and health by man-made alterations of 

physical and chemical characteristics of the atmosphere — including 

ozone depletion 
Ocean dumping and its effects on marine biology 



428 



74 

Lone term effects of low level pollution of air and water 

Environmental data handling systems 

Staffing and direction of Federal laboratories engaged in environ- 
mental research and development 

Correlation of EPA research and development activities with regula- 
tory functions prescribed by several statutes 

Evaluation of reorganization of EPA's Office of Research and De- 
velopment 

Development of environmental monitoring devices — including those 
for use in aircraft and satellites 

Roles of industry and universities in environmental research and de- 
velopment 

ENERGY 

Energy Research and Development Administration : 

Management and the planning, organization, direction and con- 
trol of programs 

Facilities utilization and facilities litigation 
Energy R&D programs of other Agencies — including EPA, NSF, 
NASA, and the Departments of the Interior, Agriculture, Trans- 
portation, Commerce and Defense (non-military) 
Through review of energy R&D laws : 

Public Law 93^09 The Solar Heating and Cooling Demonstra- 
tion Act 

Public Law 93-410 The Geothermal Research, Development and 
Demonstration Act 

Public Law 93-473 The Solar Energy Research Development and 
Demonstration Act 

Public Law 93-577 The Federal Non-nuclear Energy Research 
Development Act 

The Coal Research Act of 1960 
Specific hearings on the : 

Relationship of price to the development of synthetic fuels 

Methods for federal financing for research and development 
where industry is involved 

The role of research and development in developing a National 
energy resource data base 

Governmental planning for regional impacts of new energy tech- 
nologies 

The potential transfer to ERDA of energy research and develop- 
ment programs not transfered under the Energy Reorganiza- 
tion Act of 1974 
Research and development of specific technologies 

Coal : mining, waste disposal, combustion, gasification, liquefac- 
tion 

Oil shale, tar sands and petroleum and petroleum and natural 
gas recovery 

Conservation, transmission, storage and utilization 

Hydrogen, methonal and alternate fuel 

Federal power, solar, geothermal, and advanced energy concepts 
Joint investigations with other subcommittees and committees : 

Tax policies which influence energy research and development 

Land leasing aspects of geothermal energy demonstration and 
commercial utilization 



429 



SPECIAL OVERSIGHT REVIEWS 

The research and development programs of the Federal Highway 
Administration, the National Highway Traffic Safety Administra- 
tion, the Federal Railroad Administration, the Urban Mass Trans- 
portation Administration, and the Coast Guard 
Biomedical research and development and aspects of medical and 

genetic engineering 
Specific activities in civilian nuclear energy research and development,: 
Light water, high temperature gas and breeder reactors 
Controlled thermonuclear and laser fusion research 
Nmclear propulsion systems 
Nuclear safety research 

Environmental research related to nuclear energy 
Nuclear research and development facilities 
This list of specific and special oversight activities represents areas 
of Committee interest. However, it does ,not commit the Committee 
to examining each and every one, nor does it prevent the addition of 
oversight subjects determined by later events and conditions. 
Sincerely, 

Olin E. Teaqtte, Chairman. 



430 
COMMITTEE ON SMALL BUSINESS 



House or Representatives, 
Committee ox Small Business, 
Washington, D.C., February 18, i97o. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, UJS. House of 
Representatives, Washington, D.C. 
Dear Mr. Chairman: This will acknowledge and respond to your 
letter of February 13 last concerning coordination of oversight activi- 
ties of all standing committees of the House by your Committee. 

There is attached a listing by Subcommittee of the proposed over- 
sight activities contemplated during the 94th Congress. I have desig- 
nated Howard Greenberg, Executive Director of the Committee, as 
our representative to discuss this matter with your staff. 
"With kindest regards and best wishes, I am 
Yery sincerely yours, 

Joe L. Evens, Chairman. 

SUBCOMMITTEE ON ENERGY AND ENVIRONMENT 

This Subcommittee will be responsible for oversight, investigation 
and review of all problems affecting small business arising from mat- 
ters concerning energy and the environment. 

Examples of matters to be covered would be marketing of petroleum 
products, transportation of petroleum products, review of natural gas 
and synthetic fuel development, coal, nuclear energy, petrochemical 
matters, and environmental problems. 

SUBCOMMITTEE ON GOVERNMENT PROCUREMENT AND INTERNATIONAL 
TRADE 

This Subcommittee will be responsible for oversight, investigation 
and review of all problems affecting small business relating to gov- 
ernment procurement, as well as the problems of small business in in- 
ternational trade. 

Examples of the matters to be covered would be procurement under 
regular government procurement programs. Small Business Act 8(a) 
procurement by SBA, set-asides, subcontracting, renegotiation of gov- 
ernment contracts, and government regulations involving small busi- 
ness procurement problems. 

SUBCOMMITTEE ON SBA OVERSIGHT AND MINORITY ENTERPRISE 

This Subcommittee will be responsible for the investigation and 
review of SBA operations and of matters affecting minority business 
enterprise. 

(76) 



431 



77 

SUBCOMMITTEE ON ACTIVITIES OF REGULATORY AGENCIES 

This Subcommittee will be responsible for oversight, investigation 
and review of all problems affecting small business relating to con- 
centration, monopoly and other matters involving regulatory agencies. 

Examples of the items to be covered would be unfair and deceptive 
trade practices, advertising techniques, credit regulation, monopolistic 
practices, and antitrust and anticompetitive practices. 

SUBCOMMITTEE ON COMMODITIES AND SERVICES 

This Subcommittee will be responsible for oversight, investigation 
and review of all activities affecting small business concerning prod- 
ucts, commodities, and services, except for matters relating to energy. 

Examples are services industry problems, product and commodities 
problems, grain marketing and transportation, freight forwarding 
and customs brokerage, franchising, red tape and paperwork burdens 
of small business, and similar matters. 



432 
COMMITTEE ON STANDARDS OF OFFICIAL CONDUCT 



U.S. House of Representatives, 
Committee on Standards of Official Conduct, 

Washington, D.C., February lJf, 1975. 
Hon. Jack Brooks, 

Chairman, Committee on Government Operations, House of Repre- 
sentatives, Washington, D.C. 
Dear Jack : This is in reply to your letter of February 13, 1975, 
inquiring about planned oversight activities by the Committee on 
Standards of Official Conduct, pursuant to Clause 2(c) of Rule X of 
the House. 

At this time, our Committee plans to continue its oversight activities 
in the two areas in which it has primary jurisdiction : administration 
and enforcement of the Code of Official Conduct under Rule XLni 
and Financial Disclosure under Rule XLIV. 

Our staff director, John M. Swanner, will be available to confer 
with your staff on this matter at any mutually agreeable time. 
With warm regards, I am 
Sincerely, 

John J. Flyxt, Jr., Chairman. 
(78) 



433 
COMMITTEE ON VETERANS' AFFAIRS 



U.S. House of Represk natives, 
Committee on Veterans' Affairs, 
"Washington, D.C., February 19, 1976. 
Hon. Jack Brooks. 

Chairman, t'o?nniittet on Government Operations, House of Repre- 
s ( ntatives, Washington, I>.C. 

Dear Mr. Chairman: This is in response to your letter of Feb- 
ruary 13, 1975 calling attention to the report that is required "60 days 
after the Congress convenes". We are planning oversignt activities as 
follows: 

1. Inquiry into the operation of the military PREP, tutorial and 
work-study' programs, which are part of the veterans' education and 
training program. Particular attention will be given to substantial 
overpayments which are developing in the education and training 
program. 

•1. The Committee is continuously involved in oversight of the hos- 
pital and medical program. We have just completed a survey of prob- 
lems existing in the system. We will continue investigating complains 
relating to care, and we are giving special attention to the program 
of the Veterans' Administration for the furnishing of certain prescrip- 
tion drugs. We also monitor the Veterans' Administration hospital and 
medical construction budget which exceeds $300 million. 

'.). Comprehensive oversight is planned in the cemetery field with re- 
gard to adequacy and need for expansion and renovation of the Na- 
tional Cemetery System. 

4. Oversight investigations are underway regarding the impact of 
termination of the '"Vietnam era" on veteran benefits. 

5. Several years ago the guaranteed loan program was extended to 
mobile homes; oversight of the functioning of that program is 
planned. Late last year Congress extended the guaranteed loan pro- 
gram to condominiums, and as that program develops, oversight ac- 
tivities will be necessary. The Committee investigates complaints 
which it receives regarding poor construction and bad sales or real 
estate practices relating to the veterans' home loan program. 

fi. Oversight studios arc in progress regarding income levels of needy 
veterans receiving pensions. Studies are also underway to make the 
non-scrvice-connected pension program more responsive to the acute 
medical needs of the pensioner. 

Mr. Oliver E. Meadows, Staff Director, will serve as liaison with 
your staf:' representative and we look forward to cooperating with 
your Committee. 
Sincerely. 

Ray Roberts, Chairman. 
(79) 



434 
COMMITTEE ON WAYS AND MEANS 



11 1 /- ' U.S. House of Representatives, 

; •'• . . Committee on Wats and Means, 

Washington, D.C., March 6, 1975. 
Hon, Jack Brooks, 
Chairrrvah, 

Committee on Government Operations, 
U.S. House of Representatives 

Bear Mr. Chairman : This letter is a follow-up on your letter and 
the meeting between your staff representatives and the Chief Counsel 
and Minority Counsel of the Committee on Ways and Means concern- 
ing proposed oversight activities of the Committee on Ways and 
Means, to comply with the new rules of the House for the initial re- 
port on that subject by your Committee. 

In view of the requirements of H. Res. 988, 93rd Congress, that 
each Committee of the House establish at least four subcommittees 
and the expanded requirements of the House rules as to oversight by 
the Committees of the House, the Committee on Ways and Means has 
been extensively restructured in order to meet those requirements. 
Thus, prior to the close of the 93rd Congress, the Committee created 
six new subcommittees, as follows: (1) Subcommittee on Social Secu- 
rity; (2) Subcommittee on Health ; (3) Subcommittee on Unemploy- 
ment Compensation; (4) Subcommittee on Trade ; (5) Subcommittee 
on Public Assistance; and (6) Subcommittee on Oversight. The full 
Committee has reserved jurisdiction over the Internal Revenue Code. 

In establishing these subcommittees, it was the feeling of the full 
Committee that oversight activities should not be exclusive but should 
be concurrent, that is, that the substantive subcommittees should ex- 
ercise oversight on matters within their particular jurisdiction and 
that the Subcommittee on Oversight likewise should have oversight 
responsibility. To this end, the rules of the Committee provide that : 
"the jurisdiction of the Oversight Subcommittee shall include all 
matters within the scope of the Committee but shall be limited to 
existing law and said oversight jurisdiction shall not be exclusive but 
shall be concurrent with that of the other subcommittees and with 
respect to matters involving the Internal Revenue Code said con- 
current jurisdiction shall be shared with the full Committee." 

The Committee has submitted its budget and we are hopeful that 
in due course it will be approved by the House so that staffing can 
proceed for the various subcommittees. Clearly, in view of the re- 
structuring of the Committee and the responsibilities of the subcom- 
mittees, if staffing proceeds in due course, the Committee on Ways 
and Means will be in a position to exercise vigorous and extensive 
oversight over all the matters within its jurisdiction. 

(») 



435 



81 



It is contemplated that the Committee and its subcommittees will 
not only provide continuing oversight over the general matters within 
its jurisdiction but that, through its subcommittees and through the 
Subcommittee on Oversight, it will provide extensive oversight into 
specific areas of concentration in depth. Thus, while the Committee 
will exercise a continuing general oversight over the Internal Revenue 
laws, and the Subcommittees will carry out their continuing responsi- 
bilities over the Social Security Act, the customs and trade laws, 
health programs, welfare programs and unemployment compensa- 
tion, we anticipate that the Subcommittee on Oversight and each of 
the Subcommittees will be selecting specific subjects for very careful 
and deep analysis as time permits. 

We anticipate that in social security the following will be involved : 
(1) administration and services provided by social security district 
officers; (2) social security disability program ; and (3) social security 
computer programming; in welfare: (1) evaluation of quality con- 
trol program by HEW; (2) administration of SSI program: and 
(3) monitoring implementation of new social service legislation: in 
health: (1) the PSRO program; (2) the implementation of the renal 
disease program; and (3) nursing home care: in trade: (1) review of 
the collection and publication of import statistics; (2) preparation 
for and conduct of negotiations under the Trade Act of 1974 ; and in 
unemployment compensation: (1) operation of the special unemplov- 
ment assistance program; (2) operation of the State agencies. The 
Oversight Subcommittee has already met with the Commissioner of 
Internal Revenue and has requested a considerable amount of data 
with regard to specific matters concerning the administration of the 
internal revenue laws. We expect that the oversight in this regard will 
be coordinated with activities of the Joint Committee on Internal 
Revenue Taxation and with the Internal Revenue Subcommittee of 
the Senate Finance Committee. 

In view of our heavy schedule, the subcommittees have not de- 
veloped complete agenda as to the specific areas of concentration, 
although the Subcommittee on Oversight has set forth a number of 
possible subjects of inquiry, too numerous to set forth here, to be co- 
ordinated with the work of the other subcommittees. It should be 
understood that the Committee and subcommittees will not necessarily 
be limited to the above listing. 

We appreciate this opportunity to advise you of these activities of 
our Committee. 
Sincerely, 

Al Ullman, Chairman. 



436 



APPENDIX B 



OVERVIEW OE FEDERAI AGENCY PROGRAM EVALUATION ACTIVITIES 
JOINT INQUIRY BY THE OFFICE OF MANAGEMENT AND BUDGET 
AND THE GENERAL ACCOUNTING OFFICE 



Introduction 

The Office of Management and Budget (OMB) and the General Accounting 
Office (GAO) share a co-nmon requirement to gather certain basic informa- 
tion on the activities and output of program evaluation units within the 
Federal government. This questionnaire has been developed jointly by OMB 
and GAO to meet the information needs of both groups. OMB's main interest 
is in updating and expanding their current file of descriptive information 
on Federal evaluation activities, which was begun in 1974 by their Evalu- 
ation and Program Implementation Division, focusing on: 

--Identifying organizational settings and priorities of units for 
which program evaluation represents a major and continuing 
responsibility; 

--Discerning the nature of operating relationships between evaluation 
units and units involved in related information producing activities 

--Describing the processes employed to plan, coordinate, and dissem- 
inate evaluation studies; and 

--Obtaining evaluation expenditure and resource information. 

The primary interest of the GAO in this joint activity is in respond- 
ing to requirements of the Congressional Budget Act of 1974. This act 
gives GAO responsibilities, shared with the OMB, the Congressional Budget 
Office, and the Secretary of the Treasury to: 

1. Develop, establish, and maintain an up-to-date inventory and 
directory of sources and information systems containing fiscal, 
budgetary, and program-related data and information, and a 
brief description of their content. 

2. Provide, upon request, assistance to committees, joint commit- 
tees, and Members of Congress in securing Federal fiscal, budg- 
etary, and program-related data and information from the sources 
identified in such inventory and directory. 

The basic data gathered as a result of this inquiry will be organized 
into a pilot file system for evaluative data on agency programs by the 
Congressional Information Services Group, Financial and General Management 
Studies Division of GAO, for use in responding to congressional requests. 
OMB will utilize this same body of information to examine the way in which 
evaluation efforts are structured and managed, and how the results of eval- 
uation studies are disseminated. 



2/75 



437 



Personnel from OMB and GAO will work closely in arranging for inter- 
views with agency officials and will exchange the basic data collected to 
minimize the demands on the time of agency evaluation staffs. It is rec- 
ognized that some time will be needed by these staffs to collect certain 
information. Return visits will be made as needed for clarification of 
questions or understanding of information provided. 



Defini ti on of E valuat ion 

For the purpose of this inquiry, evaluation is defined as relatively 
structured, systematic analysis of actual performance of operating pro- 
grams, and comparison of alternative programs, to assess the impact or 
effectiveness in attaining stated goals or objectives or to assess their 
efficiency. 

Within this broad definition are four general categories of evalu- 
ation, delineated generally on the basis of their purpose and predominant 
methodology. These are; (1) substantive impact evaluations, (2) relative 
effectiveness evaluations, (3) process or management evaluations, and 
(4) project evaluations: 

1. Substantive impact evaluations attempt to measure the impact which 
Federal programs have upon their stated objectives. This type of 
evaluation seeks to determine what the program accomplishes, how 
these accomplishments compare to their intended purposes, and 
their costs. The purpose of such evaluations is primarily to pro- 
vide information for use in major policy formulation. 

2. Relative effectiveness evalua tions seek to compare the effective- 
ness of two or more major program strategies or approaches in 
attaining ultimate objectives within a national program. These 
studies are designed to help policy officials and program managers 
select the most effective mix of services to maximize programs' 
total impact, such as the mix of skill training, remedial educa- 
tion, and job search assistance in a manpower program. However, 
these studies do not necessarily measure the impact of the total 
program in absolute terms on its objectives. 

3. Process of manageme nt e valuation s are designed to measure the oper- 
ating efficiency of national programs. They are intendad princi- 
pally to help program managers achieve the most efficient deployment 
of available resources, rather than help policy officials arrive at 
major decisions affecting the scope and focus of the national 
programs . 

4. Project e valuations may entail any of the three preceding types 
(substantive impact, relative effectiveness, or process evalua- 
tion) as well as project rating -- comparing the effectiveness of 
of one or more individual projects against others. Project eval- 
uations are directed to individual, locally- based projects which 



2/75 



438 



are components of a national program, and are not necessarily 
intended to yield conclusions regarding the impact or efficiency 
of the total national program. 



Exclusions 

For the purpose of this inquiry, the following information producing 
activities, although related, are excluded: 

--Routine policy analysis and budget review; 

--Basic research not directed toward program outcomes; 

--Data collection; and 

--Audits for legal or fiscal compliance (however, audits that deal 
with program results or impact are of interest to us). 

Definition o f an Evaluation Unit 

Within each department or agency being interviewed, a number of dis- 
tinct organizational units may be performing program evaluation as described 
above. Each agency may have a different organizational structure for assign- 
ing evaluation responsibilities. It is desired to: 

--Identify and interview the head of any unit that has responsibility 
for evaluating the entire range of agency programs. 

--Identify and interview the heads of any units having responsibility 
for evaluating portions of the entire range of agency programs. 

This inquiry is directed to those organizational units for which 
evaluation activities represent a primary if not sole responsibility. 
This may be a separate office of a department or agency, or a visible 
organizational unit. 



2/75 



439 



OVERVIEW OF FEDERAL AGENCY PROGRAM EVALUATION ACTIVITIES 
INTERVIEW GUIDE 

Date: I i u- Number: 

Agency/Department: 

Unit: 

Respondent : Phone : 

OMB Staff: Phone: 

GAO Staff: Phone: 

Person Completing Form: Phone: 

For questions or additional information contact- 

OMB GAO 

Stefan A. Halper Wallace M. ohen, Vssistant Director 

Rm. 10233 Pm. 500 IB, Arthur Bldg. 

Executive Office Bldg. USGAO - FCMSD 

Office of Management and Budget 441 G St.,N.W. 

Washington, D.C. 20503 Washington, D.C. z05-t6 

Tel. (202) 395-3612 Tel. (202) 376-5334 

I. General Questions on Definitions 

1. What is the evaluation unit's definition of program evaluation? 



2/75 



440 



File Number: 



2. Are there any substantial differences between this definition 
and the definition we set forth in the introduction? 



3. Does this unit perform one or ^iore of the four general categories 
of evaluation set forth in the introduction? If so, which ones? 



2/75 



441 



File Number: 

II. Organization of Evaluation Activities 

1. Name of unit : 

2. Address of unit: 

3. Name and title of head: 

4. Telephone: 

5. Office to whom this unit reports: 

6. Provide an organizational chart situating the evaluation unit 
being interviewed with respect to other evaluation units and 
with respect to agency structure. Circle or otherwise designate 
evaluation units. 



2/75 



442 



File Number; 



Provide the names of other program evaluation units within the 
agency that are involved in evaluating similar programs as the 
unit being interviewed. Identify any field components and the 
office to whom they report. 



Where field components have been identified, how is control 
exercised? (e.g., setting of evaluation priorities, evaluation 
scope, and quality control.) 



2/75 



443 



File Number: 



9. Does the evaluation unit being interviewed exercise any coordi- 
nating function with respect to the activities of the other 
evaluation units of the agency? If so, how is this done? 



10. What are the evaluation unit's missions and areas of responsibility' 



2/75 



444 



File Number: 



11. Does the unit perform evaluation support activities? (e.g., main- 
tenance of evaluation indexing or cross referencing systems.) 



12. Is there any formal connection between this unit and (1) the 

budget preparation process, or (2) the Management by Objectives 
(MBO) process? If so, describe the relationship. 



13. In general, what proportion of this unit's evaluation work requires 
the collection of new information from the public? Circle best 



estimate. 

0% 25% 50% 75% 100% 



775 



445 



File Number: 



III. Delineation of Evaluation Unit's Program Responsibilities and Functions 

Attached is a listing of programs reported for your agency in the 
1974 Catalog of Federal Domestic Assistance Programs . 

1. Does the unit being interviewed know of any additional program 
titles that should be added to this list in order that we have 
a more complete listing of programs administered by the agency? 
For each additional program not listed in the Domestic Catalog, 
provide the following: 

A. Title of program 

B. Authorizing legislation (i.e., short title, U.S. Statute, 
Public Law, and U.S. Code) 

C. Brief statement of program objectives 

D. 13-digit budget account identification code(s) 

E. Any agency identification codes for this program 



446 



File Number: 



2. Reviewing the listing of programs reported in the Domestic Catalog 
and any additional program titles supplied for item III-l above, 
indicate the programs for which the unit being interviewed has 
primary responsibility for evaluation 



2/75 



447 



File Number; 



3. For which programs listed in the Domestic Catalog and any 

additional titles supplied for item III-l above, is evaluation 
authorized? 

Legislation Authorizing Eval- 
Domestic Catalog Number or Program Title uation (Short Title, U.S. Stat.) 



2/75 



448 



File Number: 



4. For which programs reported in the Domestic Catalog and any 
additional program titles supplied for item 1II-1 above, are 
specific evaluation studies or reports required? 

Legislation Requiring Evaluation 
Domestic Catalog Number or Program Title (Short Title, U.S. Statute etc.) 



2/75 



449 



File Number: 



IV. Evaluation Plans and Products 

1. Describe the process by which evaluation activities are planned 
and approved. 



2/75 



450 



File Number: 



2. Attach copy of any formal written plans describing evaluation 
activities for Fi 75. If not available, describe evaluation 
plans for FY 75. 



2/75 



451 



File Number: 



For ma jor studios in process for FY 75, please list and indicate 
estimated dates of completion. (In general, ma jor is taken to mean 
studies requiring more than 57 of the unit's resources.) 

Estimated Date 
Tit le of Completion 



2/75 



452 



File Number: 



4. What are the major areas of study planned for FY 76? (i< e., studies 
requiring more than 57. of the unit's resources.) 



2/75 



453 



File Number: 



5. Is there a formal dissemination plan for the distribution of 
evaluation reports, i.e., how are recipients determined? 



6. Are any summary collections, digests, master lists, or other index 
publications available which indicate: (1) evaluation activity in 
progress and/or (2) completed evaluation reports? If possible, 
provide copies . 



2/75 



454 



File Number: 



What are the titles of evaluation reports- produced by the agency 
in FY 73, 74, and 75? For work completed under contract, identify 
the contractor and contract number. To the extent that this infor- 
mation is provided for item 1V-6 above, a repeat or reformated 
listing is not required. The following elements of information 
should be provided: 

A. Program evaluation report title 

B. Author (s) 

C. Publication date 

D. Issuing agency 

E. Agency and retrieval system accession codes (e.g., NTIS 
accession number) 

F. Titles of programs that the report evaluates 

G. Titles of programs to which the report is relevant 
H. By an asterisk or other clear indication, note any 

evaluation which requires more than 57=, of the unit's 
budget for the fiscal year 



2/75 



455 



File Number: 



V. Agency Evaluation Budgetary and Resource Information 

The following questions are directed towards determining the amount 
of funds expended by the evaluation unit in FY 73, 74, and 75 and in 
having an estimate of the manpower directed specifically towards 
evaluation. In the event that the data requested is not readily 
available, a best estimate is acceptable. 

1. Agency or department: 

2. Bureau, office, administration: 



3. Evaluation unit 

4. Respondent (s): 



Phone: 



Budgetary Data 



FY 73 



FY 74 



FY 75 



5. Budget Authorization ($000) 
for this unit 



6. Obligations for 
this unit 



($000) 



A. Salaries 




($000) 


B. Contract 




($000) 


services 






C. Other 




($000) 


7. Obligations 




($000) 


directed sol 


ely 




towards eval 


uation 




activities 






Personnel Resources 





FY 73 



FY 74 



FY 75 



8. Total professional staff 
for this unit 

9. Professional man-years 
directed specifically 
towards evaluation 

10. Total clerical staff 
for this unit 



11. Clerical man-years 
directed specifically 
towards evaluation 



2/75 



456 



File Number: 
Continuation Page -- Comments 



2/75 



457 



Jtebruary 1975 



APPENDIX C 



GAP Reports Which Include Evaluation by GAP 
or Which Deal With Agency Evaluation Matters 



B-11U37U 
B-UU87U 
B-11837P 

B-11365U 

B-125053 

B-125053 
B-130515 
B-130515 

B-130515 
B-130515 

B-1^30l'+ 

C-133182 



Tate 
Aug. 6, 1973 

Oct. 30, 1973 

March 8, 197U 

March 25, 197U 
Feb. lU, 197U 

Pet. 5, 1973 

Feb. 20, 1973 

March 1. 1973 

April 2, 19? 1 * 
May 8, 1973 

July 3, 1973 

Dec. 5, 1973 



Airmail Improvement Program 
Objectives Unrealized 

Observations on the Preferential 
Mail System 

Improved Federal Efforts Needed to 
Equally Consider Wildlife Conserva- 
tion With Other Features of Water 
Resource Developments 

Telling America's Story to the World — 
problems and Issues 

More Intensive Reforestation and 
Timber Stand Improvement Programs 
Could Help Meet Timber Demand 

More Usable Dead Or Damaged Trees 
Should be Salvaged to Kelp Meet 
Timber Demand 

Difficulties of the Neighborhood 
Youth Corps In-School Program and its 
Management Problems 

Activities of the Emergence Foci ur;d 
Medical Services Project Alabama 
Council on Human Relations, Lee County, 
Alabama 

Restructured Neighborhood Youth Corps 
Out-of-Scheol Program in Urban Areas 

Evaluation of the Office of Economic 
Opportunity's Performance Contracting 
Experiment 

Industrial Management Review of the 
Naval Air Rework Facility, Alameda, 
California 

Reemployment Assista.^3 for Engineers, 
Scientists, and Technicians Unemployed 
Because of Aerospace and Defense Cutbacks 



458 



Jtebruary 1975 



Number 


Date 


B-133283 


July 30, 1973 


B-IU6285 


April 25, 197U 


B-li+6879 


Jan. 30, 1973 



B-1U6998 
B-151261 

B-159896 

B-159896 

B-16171+0 
B-163058 
B-163375 

B-163375 

B-163375 
" -163375 

B-163375 



Feb. 27, 1973 
Nov. 30, 1973 

April 11, 197U 

Dec. 17, 1973 

Nov. 23, 1973 
.Tune ?0 . ] 9"3 

April k, 197U 

Sept. 2S, 1973 



Jan. 30, 1973 

Jan. 31, i;-~3 
March 15, 1973 



Title 

U.S. Foreign Aid to Education: 
Does Brazil Need it? 

Revenue Sharing: Its Use by and 
Impact on Local Governments 

Study of Federal Programs for 
Manpower Services for the Dis- 
advantaged in the District of 
Columbia 

What Should U.S. Policy be for 
Development Assistance to Ecuador? 

Propriety of Minimum Wage Deter- 
minations for Clerical and Other 
Office Employees Under the Service 
Contract Act 

An Industrial Management Review of 
the Maintenance Directorate, 
San Antonio Air Materiel Area, 
San Antonio, Texas 

Industrial Management Review of the 
Army Aeronautical Depot Maintenance 
Center, uorpus Cnristi, Texas 

Agricultural Program Evaluation 
Laws and Studies 

Army Air i>et>ns<?: Tn«? -fAM-D 
Program 

Administration of Small Business Loan 
Program Under the Occupational Safety 
and Health Act 

Slow Progress Likely in Development 
of Standards for Toxic Substances and 
Harmful Physical Agents Found in 
Workplaces 

Occupational Safety and Health Standards 

Occupational Safety and Health Standards 

More '^ncerted Effort Needed by the 
Federal Government on Occupational 
Safety and Health Programs for Federal 
Employees 
2 - 



459 



February 1975 



Report 
Number 


Date 


B-1637U2 


March 21, 1973 


B-16307U 


May 3, 1973 



B-163922 

B-163922 
B-163922 

B-163922 

B-163922 

B-163922 

B-x6403± (1) 

B-16U031 (2) 
B-16U031 (3) 
B-I6U031 (3) 

B-16U031 (3) 



Sept. 23, 197U 

Aug. 1, 1973 
March lU, 1973 

June 8, 1973 

Feb. 25, 197U 
March 29, 197^ 

March 7, 197U 

March 11, 197*+ 
April 3, 1973 
June 27, 1973 

June 13, 1973 



Title 

Use of Excess Defense Articles and 
Other Resources to Supplement the 
Military Assistance Program 

Advantages and Limitations of 
Computer Simulation in Decision- 
making 

Opportunity to Increase Effectiveness 
of JOBS -Type Programs to be Funded 
Under CETA 

Public Employment Programs in 
Selected Rural and Urban Areas 

Impact of Grants to Indian Tribes 
Under the Emergency Employment Act 
of 1971 

Public Service Benefits From Jobs 
Under the Emergency Employment Act 
of 1971 

Cleveland Summer Youth Employment 
Programs 

The Emergency Employment Act: 
Placing Participants in :;onsubsidized 
Jobs and Revising Hiring Requirements 

Problems of the Upv/ard Bound Program 
in Preparing wisadvsnt.agfivi Sfvudehts 
for a Postsecondary Education 

Progress and Problems in Providing 
Health Services to Indians 

Effectiveness of Vocational Rehabili- 
tation in Helping the Handicapped 

Social Services: Do They Help Welfare 
Recipients Achieve Self-Support or 
Reduced Dependency? 

Some Problems in Contracting for 
Federally Assisted Child-Care Services 



460 



February 1975 



Report 
Number 

B-16U031 (3) 



-16U031(3) 



B-16UU97 (3) 



B-166506 



B-166506 



B-171019 



3-171019 



B-171019 



B-171526 



B-171630 



Date Title 

June 27, 1973 Problems in Administering Two 

Eligibility Aspects -- Incapacity 
and Unemployment — In the Aid to 
Families With Dependent Children 
Program in Pennsylvania 

Nov. 20, 1973 Evaluation of Efforts to Determine 
Nutritional Health of the U. S. 
Population 

April 2k, 1973 For Safer Motor Vehicles — More 

Effective Efforts Needed to Insure 
Compliance With Federal Safety 
Standards 

Nov. 15, 1973 Improved Federal and State Programs 
Needed to Insure the Purity and 
Safety of Drinking Water in the 
• United States 

Jan. l6, 197^ Research and Demonstration Programs 
to Achieve Water Quality Goals: 
What the Federal Government Needs 
to do 

Oct. 21, 197*+ Progress in Determining Approaches 
Which Work in the Criminal Justice 
System 

Jan. l6 3 1973 Development, of a Nationwide Criiiunai 

Data Exchange System — Need to 
Determine Cost and Improve Reporting 

March 19 , I97U Difficulties of Assessing Results 
of Law Enforcement Assistance 
Administration Projects to Reduce Crime 

May 22, 1973 Interim Report on the Agency for 

International Development's Housing 
Investment Guaranty Program 

March 28, 197*4- Observations on Housing Allowances 

and the Experimental Housing Allowance 
Program 



k - 



461 



February 1975 



Report 
Number 


Date 


B-171630 


March 27, 197U 


B-172707 


Aug. 13, 1973 


B-175155 


Aug. 22, 1973 


B-17601+9 


Jan. 22, 1973 


B-177011 


Aug. lU, 1973 


B-177166 


June 12, 1973 


B-177952 


May 2, 1973 


9-178561* 


June 29, 1973 


B-1737U1 


Nov. 29, 197U 


(KED-75-31C) 


Dec. 18, 197- 



Title 

Problems in the Homeowner ship 
Opportunities Program for Low- 
Income Families 

The .Army Reorganization for the 1970s: 
An Assessment of the Planning 

Railroad Reservation, Information 
and Ticketing Services 3eing Improved 

Need to Improve Language Training 
Programs and Assignments for U.S. 
Government Personnel Overseas 

Federal and State Efforts to Control 
Water Pollution Caused by Acid Drain- 
age From Mines 

In-FIight Escape Systems for Heli- 
copters Should be Developed to Prevent 
Fatalities 

Problems in Meeting Military Manpower 
Needs in the All-Volunteer Force 

Progress and Problems in Achieving 
Objectives of School Lunch Program 

Employment Services for Vietnam — 
Era Veterans Couxd be Improved 

Special Supplemental Food Program 
Food and Nutrition Service 



- 5 - 



462 

■ LiLlll-JJzlL ' ' ''iwal Act ■oiiiit i 1 1^ Ol f ic< - Reports 
Involv in^ I' ro^r a in Evaluation 

In its reviews of a variety of Federal programs, t lie General 
'counting Office has conducted evaluations and lias reviewed the validity 
of agency evaluations. Although problems have been found and reported, 
the overall tone has been positive regarding the potential value of 
program evaluation, as the following selected examples show: (Underscoring 
to emphasize positive points.) 

1. In Observations on the Preferential Mall System , B-114874, 
October 30, 1973, GAO made use of a computer model to calculate 
alternative costs. The report explains the reasons for our corrections 
to what we believe to be "questionable" assumptions. 

2. In Social Services; Do They Help Welfare Recipients Achieve 
Self-Support or Reduced Dependency ?, B-164031(3), June 27, 1973, we 
stated that we used an approach which could give case workers a more 
systematic and analytic means to assess recipients potential and, 
accordingly, which could improve the allocation of social service 
resources. We stated that this approach was developed by the Denver 
Department of Welfare and our report contains a chapter dscussing this 
method and concluding that it is feasible to develop a predictive 
approach using recipient characteristics and that further refinement 
of the approach could be the next step . 

3. In Evaluation of the Office of Economic Opportunity's 
Performance Contracting Experiment , B-130515, May 8, 1974, GAO 
demonstrates the capability to assess a social experiment which included 
a built-in evaluation design. This can viewed as positive with regard 
to evaluation because the problems reported are in the implementation 
more than in the design. The digest states that observations contained 
in this report are expected to be of value to the Institute (National 
Institute of Education) and local educational authorities if similar 
experiments are conducted in the future. 

4. In Maternal and Child Health Programs Authorized by Title V , 
Social Security Act , B-164031(3), June 23, 1972, on program evaluation 
systems for these projects which points out some problems in implementa- 
tion but ends on a positive note "on the basis of our review of the 
system's general design, we believe that it is capable of producing 
information useful to program managers in measuring project and program 
effectiveness." The report contains specific recommendations for actions 
which might improve the system. 



463 



5. In Improvements Needed in the Administration of Contracts 

for Evaluations and Studies of Antipoverty Programs , B-130515, December 28, 
1971, problems are stated and specific recommendations are given for 
Improving both contracts and monitoring, to develop and implement a 
system to Insure the formal assessment of the results of the evaluations 
and studies, and the effective utilization of the results of contractors' 
studies to Improve the design and administration of programs or to 
develop new program approaches when appropriate. 

6. In Need for Improving the Administration of Study and 
Evaluation Contracts , B-164031(l), August 16, 1971, containing problems 
and recommendations with regard to education program evaluations similar 
to the report on antipoverty programs (see above item). 

7. In Observations on Housing Allowances and the Experimental 
Housing Allowance Program , B-171630, March 28, 1974, GAO has highlighted 
some of the issues involved in evaluating such a program including 
studies conducting by a GAO consultant. GAO recommends that the Congress 
in considering future legislation authorizing a national housing allowance 
program, weigh the benefit that could be derived from waiting until the 
experimental program is complete and more information is available on 

the likely Impact of such a program. Thus, this report is a positive 
point of review regarding the benefit of evaluation and of waiting for 
such evaluation results for use in major decisions . This report is 
similar to the staff study and testimony provided to the Senate Pinance 
Committee in 1970 on evaluation of the New Jersey negative income tax 
experiment. 

8. In Progress in Determining Approaches which Work in the 
Criminal Justice System , B-l 71019, October 21, 1974, GAO recommends 
that the Law Enforcement Assistance Agency should issue guidelines 
requiring state plans which discuss how evaluation will be used to 
assist in management decisions and the extent to which administrators 
believe current evaluation strategies need to be modified to make this 
possible. The report also states that there is no doubt that outcome 
evaluation is complicated and in some instances costly. The 
consequence of not doing such evaluations, however, is to reduce the 
planning process to chance. Evaluations are necessary so more 
objective decisions can be made regarding allocations of resources. 

9. In Difficulties of Assessing Results of Law Enforcement 
Assistance Administration Projects to Reduce Crime , B-171019, 

March 19, 1974, GAO recommends designation of demonstration projects 
to determine information that should be gathered and the type of 
evaluation that should be done and the standardized methodologies and 
standard data reporting systems to accomplish this. 



464 



APPENDIX D 



Program Evaluation: A Manual for Legislators 
and Legislative Staffs 



Draft of January 1975 



Prepared by William H. Foskett, Consultant, and Harrison W. 
Staff of Senator Bill Brock, (Tennessee) 



Comments on the draft are invited and should be sent to 
Harrison W. Fox, Jr. Office of Sen. Bill Brock, 
254 Russell Senate Office Building, Washington, D.C. 20510 



465 



INTRODUCTION 

Finding and usinr the links between legislative r.onls, govern- 
ment spending, program activity, and consequences are at the heart 
of Congressional authority. Failure to make strong, valid, links 
Invites the consequences of an uncontrolled budget and/or unsolved 
human and social problems. The Legislative Reorganization Act of 
1970 and the Congressional Budget and Impoundment Control Act of 
197*4 in principle provide Congress with an information resource 
mechanism and a budget procedure necessary "to exert its power of 
the purse with full knowledge of the consequences of its actions". 

IT Congress wants to exert its full authority under these 
Acts it will have to consider the human and social consequences, 
as well as the economic consequences, of its actions. Practically 
speaking, this will require Congress to obtain and use evidence 
which links goals, spending, program activities and program results 
to each other. In setting national priorities and working within 
self-imposed budget limits Congress will need solid evidence 
about the cause-effect relationships between program results and 
Congressional goals. Cause-effect relationships can be verified 
only within the logic of experimentation , (e.g. the negative 
income tax experiment), thus Congress will need to provide for such 
experiments as it seeks answers to questions about what services 
will cause the results it seeks. 

Causality is not the substance of every legislative question. 
So Congress will also need to use other non- experimental evaluation 
methods — e.g., needs assessment. 



466 



We aim to provide legislators and their staffs with sufficient; 
knowledge of evaluation research to maintain their political 
autonomy while focusing on the best possible evidence Jfcr the legi- 
slative questions at hand. This manual does not aim to make the 
reader into an instant expert on highly technical matters of evalu- 
ation. Evaluation users should keep in mind that serious limitations 
currently exist in the organizational resources available to supply 
such evidence and in the state of the art of evaluation research. 

By taking the steps listed below you can build a sound evidence 

base for deciding how to vote on authorizations and appropriations. 

In addition, you can build an evidence base for strong committee 

oversight of legislation. Each of these steps will be described 

in the text of this manual. 

Step 1: Selecting questions to be answered and evaluation 
techniques to be used, 



Step 2 
Step 3 
Step H 
Step 5 
Step 6 
Step 7 



Drafting legislative language. 
Formulating program goals and terms 
Measures. 
Design. 

Statistical analysis. 
Format for evaluation studies. 



Step 1: Selecting questions to be answered and evaluation tech - 
niques to be used . There are some simple general types of questions 
which have profound practical meaning for effective legislation. 
Particular instances of these general questions being asked are 
common to most legislation at some phase of its life. Such questions 
are listed across the top of Table 1. There is at least one or 
more evaluation technique (s) which can supply evidence in response 
to each of these questions. A basic list of the categories of 
these techniques is shown down the left side of the table. 



467 



Table 1 
(Being Developed) 



LEGISLATIVE 
QUESTIONS 



Your Suggestions are welcome 



468 



Many questions will come up during the life of an Act — 
from its introduction as a bill, to its consideration for re- 
authorization. You should act to see that the type of evaluation 
research that matches these questions is beg*n early. It is 
important to note that the selection of an evaluation strategy 
must in most cases be completed before program implementation begins . 

Today, most legislatively-mandated and executive-initiated 
evaluation is not framed so as to provide answers to basic legi- 
slative questions which arise especially in oversight. Indeed 
these provisions seem to satisfy mainly executive branch interests. 
The most common Congressional control device is to be found in 
the lanfruap-o which deals with measures of effectiveness and/or 
the nature of recommendations to be made to the Congress. (Examples 
of such legislative language may be found in Program Evaluation : 
Legislative Language and a Users Guide to Selective Sources , 
Part 2. U.S. General Accounting Office, 1973.) 

Though there are many laudible features in this language, such 
provisions for evaluation have typically failed to provide Congress 
with the kind of information it needed at the time it needed it. 
Often such language has been self-defeating by blurring the purpose 
of evaluation, or by failing to authorize the necessary conditions 
(e.g., randomization") or resources to obtain the evidence. 

In the next section you w^.11 find legislative language options 
along a number of crucial dimensions which can determine evaluation 
activity. By selecting appropriate options the drafter of legi- 
slation can clearly authorize and provide an opportunity for the 
collection of evidence necessary to answer the right questions at 



the right time, with the right evaluation technique. 

Step 2: Drafting legislative language . Congress often finds 
that it doesn't have the right kind of information at the right 
time, even though an extensive "report" may have been completed. 
The simplest and most common reason for this is that the legi- 
slative language of an act's reporting sections provides little 
or no guidance for Congressional evaluation needs. The language 
of the reporting section provides a basic foundation for oversight. 
Effective Congressional oversight is to a large extent dependent 
upon the specification of a reporting section. 

If the language in the reporting section is too specific, 
Congressmen or their staff may be entangled with technicalities 
or petty details of planning or managing evaluation research. 
If the language is too vague, other Interests may pre-empt Congress 
need for timely, pertinent, and credible oversight information. 

There are eight aspects of evaluation which Congress should 
deal with in the language of reporting sections of legislation: 

- purpose(s) of evaluation 

- who will conduct the evaluation research 

- content of evaluation 

- the timing of evaluation 

- funding for evaluation 

- congressional veto 

- to whom evaluation information will be reported 

- access to information 



470 









E«+H 








c 
o 

s 




E rn 












■d 


cO O 






Cu 




rt cu 


E 


to 






13 


rH in 


r- 




rH 




4-1 


rH > 


rt 


10 CD 








o +-> 


60 CO 


o 


CU (0 


Cu 




•H 


60-H 


u 


+H +J 10 






nJ 


X -H 


O rH 


■p 


XI rH 


E 


*-> 


rH 


O rH 


60 


O C rt 










rH CO 


E 


■M Cu 


CO 


10 


rt 


rH CU 


O Cu • 


CD O 








--t-a 


tX o 


I>N CO 


C 


X 




3 


OhX) 


r- X m 


10 -H 






i— I 


rH c 


co 


U U 


f+H a. 


cu 


d 


C 




cx+j +-> 


E rH tO 








•H « 


60 


CO 60 


o o 






O 


l+J (0 


to 


rn U 3 






5 


JC 


C <D 


(/] C 






lM "^ 


+J « 


O CD 


to Uh o 


CD O 








in 


•h x 


W r-l 


C rH 


o 


■3 rt 


cu • 


o 


•rH O U 


«-> rH 3 






■M ■ 


+J C 


e-+-> 


cu f- 


O Ci> 


-H 


rt U 


in O. • 


10 -H 


X 


O 60 






rt • 


CO 0) 


o 


u 


•H rH 




O 


cu >. • 


*-' > 


♦- If, cu 


C^-H 






X ■ 


X n 


rH 60 


O 4-1 






krj rH 


u +-» • 




i- x: 


■H XI X 






+-> in 


+J -H 


cu C 


c o 


ai o 




C ^^ 


•H t/1 


CD Q) 


E o +-> 


C E 






CD 








3 rH 




rt 


> CU 10 


U-t to 


O +-" 


t/i rt rt 






C -H 


C -H 


CD +-> 


C i/i 








r- ,C O 




(- U r- 


E 






O +-• 


o u 


T3 a> 


o m 


10 & 




60 ■ 


CU +- rH 


Cu +-> 


4h CD rt 


CO l/. XI 






■H -rt 




cu 


■H Cu 


r> E 


c 


C • 


in co 


i C 


in cu 


(- *-> C 






+-> rH 


«-> o 


rn E 


4-| C 


M-l O 


o 




C c 


o cu 


in x> 


CO 'J (0 






CO H 


cfl «-> 


o 


"i <b 


u 




rd T3 


C+H +- O 


x) E 


+-> X 


O Cu 






EX> 


E 


■\ c 


E > 


-3 




c a» 


O C_J 


■h 3 


•h u XI 


rH -r-, t/, 






>- -H 


U Cu 


XI -H 


rH >H 


C. Cu 


CO 


3 *J 


ej 


u. u 


hH -H C 


Pu O Cu 






O o- 


O rH 


c 


O •(-> 


co x: 


u 


<+H CO 


>• o >. 


O 


cu x; co 


U U 






^ £ 


4h ,X 


CO Cu 


Mh O 


+j 


3 


3 


rH 10 X 


r- X) 


c s 


hh a3 






c o 


c « 




C cu 


i v - / 




CD rH 


<b -H 


o 


<d in 


O rH 






•H PL, 


■H +-> 


O0-H 


•HlH 


cu 




x: ti 


> rnT3 


■^ o 


Xi ^h +-> 








to 


c 


c +j 


tw 


■H • 




■*-> > 


•H CO CD 


X) +- 1 


O 'H 


to O CO 






CO Cu 


co 3 


•H (J 


60 CD 


> • 


l+H 


cu 


— 1 CX-d 


c 


XI <+H 








c s- 


c o 


r-, (Ll 


C 


(U 


o 


«+H 


CU £ C 


rt • 


c c CU 


<— < / — V 








•r-l (J 


kh Lh 


•H Tj 






O 30 


TO O Cu 


r^ 


rt O C 


3 C « 






XI -M 


"J u 




TJ C 




CD 


c 






• H CU 


t/. O tO 






■H -C 




4-' O 


•h re 


E U 






.H C 


♦J l-O 


1/ «-> X 


CD -H CD 






> Ofl 


> 


C 




CO T-l 


3 


C CD 


rt C -H 


(J 


+H CO 


i~ +-" to 






O -H 


o c. 


CU CU 


O >. 


U <4-l 




Q) XJ 


3 -H 


a> io 


in u CD 


■H to 






(-, I/. 


r- O 


T> X 


r-. U 


rJ<+H 




T3 




<H-I CD 


o -H x 


CU C CU 






p**-. 


pt,-i '■ 


•H 


& q 


>:> O 


*-• 


c >, 


I ) « J.' 


<»H t-. 


o Vt +-> 


> H O 






CD 






cu 


r- 


to ■ 


0) 4-" 


CU '/I o 


a> -h 




rH U, V 






<4( ' • 




l+l rH . 




D. tfl 


c. 


C--H 


«J TH 




l-H |.l P. 


*-> O 3 






o b 


O r- 




o u 






ttJ > 


Cu C > 


iD c 


r-l f, CO 


rt XI t/i 








4-> 




•H 


"h-I r-l 




X) -l 


.c; CU I-. 


X CU 


CO 0) CD 


r-i 






<b to 


Ca in 


ti> C 


CU IfH 


C H-l 




C 4-J 


*-' -H CU 


♦-" rn 


XI r- 


4) X) C 








to -r-i 


to j:: o 


lO In • 


c 




•i-H U 


CL in 




CD H 


r- C CU 






O ft 


O C 


O U -iH 


O cu • 


Cu 3 


c 


^i 


CU -H 


O 


r. rH 


r, <U 






(X 


fi,-H 


CVr-l 4H 


(X 


U O 


o 




rn IJl+H 




3 C i-i 


*-> ^~X> 






u c 


J- EH • 


>-. x co 


r- CU • 








3 cu C 


73 • 


10 rt rt 


rn 






3 -H 


3 XI • 


3 ^ rH 


3 X -M 


l+H (J 




:m x 


l/i f-H 




rt X 


O r- CU 






EX 


Pt. CO • 


Ph l/J 


0.4-> C 


u- < 


CO 


H 4-> 


C0 rH 


rt • 


CD CO 10 


P. Cu > 




<u 


to 




10 -H 


<u 


o 




Vh 


Cu T3 CD 


Zi in 


E C 


cu x rt 




00 


cu in 


tl> £ in 


CU 0) CO 


cu 4/ E 






Ol+H 


E cu > 


E cu 


■rH >>> 


~ E x: 




« 


X: 0) 


x co +-> 


x: u cu 


x: > cu 


<L CO 


11 


o 


tj a.. 


U 


O XI E 


3 




3 


•M rH 


*J r-l C 




<-> o CO 


X t- 


X. 


C 


O C rH 


O -H 


*-| 3 O 


o c x: 




bO 


co 


CO CU 


> 


r- co 




<-» 


rt! C 


*-> <l> 


*-» > 


rH C 


+-■ o 




C 


Ih c 


rH O "H 


I-, t- CD 


rH CUC 


c 




• A 


• 4->Xl 




• u o 


• O -H 




rt 


o o 


O rH rH 


o a> x 


O E CO 


- Cu 




• ''55 


• cu 


• c u 


• x x: 




J 


IX, (J 


B. P.U 


u. to +-> 


Wh-h 6 


■ <J 




• 10 


. -H Cu 


• +-> s 


















o 

*> 

€ Ol 






*-> i 






,_, 


^ 






^ 




*» <oi 






3 X 






rt 


•H 






cO 








<f. -& 






c 








C r-^ 


3C 


5c 








CD 10 CO 10 






o 




c 




O O 


s: 


feS 








rn CD CD 






•H +J 


XI 


CD 






CU H 






X) 


to «to 






mx 


CO 


£ 


<u 


in d 


> y. 






C 10 


CD to to rt 






to co 


♦J 


E (X 


E rl 


in 




! p? 


CD t- 


CD 10 


rt 4-> 


> ID CU O 






.H 


(J c 


CO O 


CO 'U 




*-> •> 


U OJ 


U 4-> 




•H U r. 






^ m 


■H 3 


t-l rH 


rH CO 


U • 


3 • 


1—1 «H 


■H > 


■H U 


in i4n 


+J u 3 to 




to 


un- 


rH O 


CO CD 


60 CO 


'oO CO 


U 60 


rt tj 




> <u 


+-> CD 


rt 3 rH 3 




K 


ci a> 


X U 


o > 


O C 


c • 


CU • 


r- rH 


rnl+H 


to C 


r-l tO H O 




o 


o > 


3 U 


t~ cu 


rH t0 


O <i> 


X <D 


O rH 


CU o 


Cu <+-( 


O CO 


O i rt 3 




♦-> 


u o 


a. co 


a, x) 


a, E 


..J — 


W ^~ 


J s 


LH "3 


LO CD 


U X 


a i In Ol 




ex 


























., o 


_._.«.. 


-a 


o 


3 


(0_ 


-pl_ 


feL^4-«° 


X 


o 


X 














i i 

r -1 c 




10 O 






o o 


o o 










w 






-1) *-> 






+-' 4-1 






ui O oj 






>. U -0 


C co 


<0 


ox 3 






U 3 3 


CD 3 




p,+j — 






CO ■- 




-c 


(-. TO 






mco 


|c rt 


V 


3 4* > 






CO o > 


o > 


E 

u 


a. o a) 






< u a 


P " 










1 • 


_s_ 


.*"•__. 








CN| 







v~- _ 


... . 


. 





..J 



471 



~ 



o u 


O 


*-> 3 


H J= 


U rH 




rt -H 


in 3 


<+* m 


<D (0 


u-i 


t- 


X 


OO u 


lH <+H 


c o 



14-1 

M 

00 <D 



3 «-> 
rt o 



h o 
a. 



> >.u 

au b 
D. 4) a; 



1- U 4J T3 c 



oo-h 
o-o 



ex 
3 



3 < 

O <-> 



C 
3 -H 

i+-( T3 



i/l C *-> 



oo en .o 

C TJ O 

o ^ 



n a> i_ -o >- 

0)T) O^- M 



j- 6 
a o 



i/i a) o 



o tA 



Cfl 10 +J +-> 



a. - fi 



U C </> <D 



DOrH 



: c c 
I 3 <u 

: -H U 



> >. C8 T3 
I lM t- O 

<+h T> O +-> 
-H O »- <U 

DEftE 



472 



C (/) r-l 

00'— >n 3 

•t-( in to O « 

ir. v *-> -a x c 

<L> <L> U if. QO 

T3 «-> < , M 



« 



C N 
00 0) 



o e x > i/> 
"- 1 u .c -a u 



— 


XV* 


IT. 




^« 


•—1 


XI 


c« 








3 




<D 








o 


2 


X 




c 




— 


V 




9) 


O0 V 


If. 






4> 




> 




> 


rt 


+-> 


10 


■|H 







u 




4> 




X ^ 


(A 


■H 


*T3 


U 


u 






5 




- 




w 


Lu 


S 






u 


D 




O 




X) 


r.: — 


(U 


u 




o 


a 


c 


X 


v— ' 


^* 


. 



*-> (U >, O 

u, .c u_ ot 

- - — 

t3 E O 



-3 1- !«*< > 

*— so |u-. o 



- O E 



Q.-W U. P.— I 3 

■C t) « (J i/l 

o -h « o a a; 



<J o 
E 

E O O 

C ~ 



X +J 

c *-> 

o 
o 

m*j o 

+JT3 

*■* C 
• h ea 
E 

E a> 
o (U *-> 



•j a. +-> 



0) 3 
•r-l VI 

o a> 



x i- x o 



o o> o c 

&,*-> a *-> o 

DUO -^ 

t- C i/i *-> D 

u o « ,1 o 
-i in (u •-< x 

-i <-> .— c >~ 

to 0) ♦-> .— ex > 

►X X — ( o o - 

*-> E 5 ^ ' 
E C 

41 o si o,«« 
o o - < 



D. O 

<u c 



E O 

E -m 

o <u 

u > 



o - u 

rD E sa 

e c 



e 
i. — - 



•_- c- co o oa 



: 



473 



A variety of legislative choices are available in each of 
these aspects of evaluation. Legislative language for the eight 
options is shown in Table 2. 
Step 3: Formulating program goals and terms . 

In this step (Step 3) and in Steps 4,5, and 6, we will deal 
with matters which aim to give you, the evaluation user , sufficient 
control over the specifics of evaluation planning to see that the 
type of evidence you need will be supplied by the evaluation. 

A description of the program in clear concrete terms is an 
essential condition for evaluation. The program goals and services 
must be described in such a way that they can be validly measured — 
even if by relatively primitive techniques. Someone, either the 
drafters of the legislation or the program planners, must produce 
a concrete description of the program which translates Congressional 
goals into measurable objectives. 

For example, "improving job opportunities for minorities" 
will not be a measurable goal, whereas "reducing the unemployment 
rate" or "increasing the percentage of minority group members in 
management, professional, and skilled labor positions" would be 
measurable goals. 

Congress has been frequently criticized by professional 
evaluators for not writing the goals of legislation in measurable 
terms. You should not leave the definition of Congressional intent 
to those responsible for the technicalities of evaluation research. 

It will often be impractical (for passage and implementation) 
to burden a bill with the level of technical detail most convenient 
for the evaluator. Such detailing may be a barrier to passage of 






474 



n hill; or If passed may lead to a program lacking sufficient 
flexibility for successful management. We believe that you can 
solve this problem by systematically defining in the legislative 
history, preferably in the committee or conference report, the 
elements of the program or by requiring that a formal program plan 
be prepared by the program planners and submitted for legislative 
review before implementation of the program can begin — i.e., the 
Congressional veto concept applied to the program plan. 

Full measurable descriptions of objectives and actions should 
be required in program plans if you intend to consider implementation 
or effectiveness of the program, 

The actual technical manpower necessary to develop these 
elements of program description in legislative or formal program 
plans may be obtained from outside contractors, Congressional 
Budget Office, Congressional Research Service, Office of Program 
Review and Evaluation, the General Accounting Office, or Executive 
Branch agencies as authorized by the Congressional Budget and 
Impoundment Control Act of 197 1 *. 

In the committee or conference report you should also state 
feared, undesirable side-effects which evaluators should look for 
and measure — in addition to the desirable results expressed in 
program goals. Opponents of a bill will often be very astute 
sources of undesirable side-effects. 

Step k : Measures . 

After formulating program goals and defining major terms, 
one must develop the valid measures for use in the program evaluation. 



475 



These measures will supply the basic body of evidence for drawing 
conclusions about program effectiveness. For Instance, n program 
to Improve highway safety might have as Its p;oal to reduc-p the 
accident rate, to reduce personal injury as well as the severity of 
personal injury, and death, to reduce the extent and severity of 
property damage or any combination of these. A question which 
still must be answered is: How are these goals to be measured? 
Such a program might have the anticipated (thus measured ) undesirable 
cost of increasing commuting time, unemployment due to inability 
to commute to work, etc. 

It is very important to choose measures in advance to be sure 
that they will supply the necessary data on which oversight should 
be based. Legislative users of evaluation should ask the following 
questions about measures when reviewing plans setting up programs 
and evaluating their effectiveness. 

(1) Are the basic ideas of the program adequately translated 
into measurable terms? (Are you really satisfied that the goals 
and objectives of highway safety are adequately measured?) 

(2) Does the measurement cover program input? Are they 
adequate to verify that the program has in fact been implemented? 
Do they provide quantitative measures of this input? 

(3) Will the data be sufficient to answer all of the important 
questions which you should raise in oversight of the program's 
effectiveness? fDo not be caught short without measures to answer 
Important highway safety questions [ ) 

CO Is each measuring: instrument sensitive enough to show the 
magnitude of differences which the program is expected to create 



476 



both services and effects? (If you are expecting to improve 
highway safety by inches, don't try to measure that improvement 
by feet . ) 

(5) On the whole, do the measurements proposed for each aspect 
of program treatments and program effects reflect the idea which 
Congress intended the program to translate into reality? (If 

the goal is to reduce highway accidents during rush hours, collect 
the data at appropriate time intervals — daily, not weekly, 
monthly or yearly.) 

(6) Do the items composing particular measurements of parti- 
cular program treatments or effects fairly sample all of the im- 
portant aspects of that particular treatment or effect? Make sure 
that all aspects of highway safety that are perceived to be im- 
portant are being measured. Are the undesirable side effects 
represented in the measures? (For instance, highway safety 
affects property, lives, insurance rates, birth rates, and food 
consumption. But probably the first three are of interest.) 

There are rarely simple, definitive answers to the above 
questions. But by raising these questions and reviewing their 
answers, the legislative user of evaluation can gain 

evidence necessary to estimate the soundness of proposed measures. 
Developing measures is only one step in the evaluation process. 
The next step the legislative evaluation user should take is to 
examine the design^ plan for collecting data on program services 
and effects. 



477 



Step 5- Design 

The evaluation design is a plan for linking measures of 
program services and results. 

Good measures and 
sound evaluation design evaluation are both necessary for effective 
program evaluation of programs. A well conceived evaluation design 
will allow effectiveness to be attributed to the program itself 
rather than to events or circumstances outside the program. 

Weakness in design and measurement tend to obscure positive 
evidence of program effectiveness . 

It is 
very much in the interest of proponents of particular programs to 
obtain the most valid measurement instruments and most powerful 
research design available in order to prevent extraneous factors 
from obscuring program effectiveness. 

While it will not usually be appropriate for Congress to 
specify in legislation the details of research design and statis- 
tical analysis, there are a number of important design features 
which are under legislative control but which in past legislation 
have often been omitted or inadvertently precluded. 

Evaluation research design is extremely complex. It requires 
many technical judgements which cannot be made responsibly by a 
layman untrained in research design. But legislators cannot meet 
their responsibility by merely surrendering the whole matter of 
validity to experts. Legislators have a responsibility to raise basic 
questions about the validity of the evidence provided by the program 
evaluation they authorize.. You can meet this responsibility by 



478 



requiring: that evaluation experts supply straightforward answers 

to the basic threats to valid evaluation found at the design stage.* 

Table 3 below states these threats in question form: 

Table 3 
QUESTIONS 

Issues In Evaluation Design Validity 

1. Will the evaluation supply evidence that the program services 
were actually delivered in the intended type, quantity and quality? 

2. During evaluation of the program can events outside the program 
blunt or exaggerate measures of program effects? (history) 

3. Could natural changes or pre-existing trends (e.g., normal 
growth or losses) that would occur regardless of the program account 
for results? (maturation) 

4. Can the process of measuring itself induce change in that which 
Is being measured? (testing) 

5. Can the measures themselves change in meaning during evaluation 
of the program? (instrumentation) 

6. If comparison groups are used, would they have changed differ- 
ently from each other even if they had not received different 
program services? (selection) 

7- Are participants selected on the basis of extreme scores on 
measures related to program effectiveness measures? (statistical 

regression) 

8. Will the evaluation show whether participants who drop out of the 
program (or different comparison groups) are alike in some way 
which would influence program effectiveness measures? (mortality) 

9. To what extent can valid evaluation results (positive or negative) 
be generalized to other possible participants or program settings? 
(interaction of testing reactive arrangements) 

10. Is variation in program services within critical range speci- 
fied for the program? (see §1 above) 

11. Are several different ways of measuring each program effect 
used so that corroborative evidence of results Is provided? 
(generalizing) 

*More detailed examination of various design consideration can be 
found in Appendix C. 



479 



Between Steps 5 (Evaluation Design^ and 6 (Statistical Analysis) 
the program evaluation strategy will be implemented. Committee 
members and staff should keep a close watch over these tactical 
activities of program evaluation. Such techniques as formal 
reporting requirements, informal contact, etc., should be used to 
keep tabs on data collection procedures, timing of evaluation 
activities, program content, and decision making needs. 

Step 6. Statistical Analysis 

Previous steps we have recommended all aim to provide the 
soundest possible raw data base for oversight of program effective- 
ness. These steps minimize the possibility that the basic evidence 
regarding program effectiveness is confounded by inadequate program 
delivery, poor measurement, or invalid research ^design. Next this 
valid data must be condensed and focused on the issues to be raised 
in oversight by use of statistical analysis. Given valid measure- 
ments made in a design framework which yields interpretable data 
there may be more than one right way to statistically analyze the 
data. 

Statistical analysis may be used to describe raw data or to 
draw Inferences about relationships and meaning of the data. Used 
descriptively statistical analyses condense large amounts of raw 
data In such a way that basic characteristics are pointed out and 
described in succinct, numerical terms. Used inferentially 
statistical analysis searches out and tests relationships between 
different measures — especially between measures of program activities 
and program effectiveness. The variety of descriptive and in- 
ferential statistical techniques used in program evaluation is so 



480 



varied and broad that judgement as to the proper use of any par- 
ticular statistical technique ±n any specific situation must be 
left to the Professional evaluator. But legislative users of 
effectiveness evaluation should take a number of precautions to 
minimize entry of bias into results at the point of statistical 
analysis — or, in cases where bias is unavoidable, to make that 
bias apparent. 

The strategy we recommend to legislative users for dealing 
with statistical analysis has two elements: 

- documentation of four types of information (cited below) 
about the statistical analyses used in effectiveness 
evaluation reports; 

- Independent, simultaneous statistical analyses of the data 
using alternative techniques. 

By documenting the following; four basic considerations for 
statistical analysis in effectiveness evaluations professional and 
public attention can be conveniently drawn to areas where problems 
have been overlooked in the past. Such documentation should be 
clearly labeled and appended to the effectiveness evaluation report. 
Of course, each such appendix would implicitly require evaluators to 
take and report certain precautions that might otherwise be over- 
looked or unreported. Each of the topics below should be documented 
in an appendix. 

a. Incomplete or missing data should be screened and treated 
in such a way as to avoid biasing the results. Because such 
procedures are basic, no chance should be taken that they might 
be skipped over in haste or economy. 

b. All statistical techniques used to work with the data should 
be documented. Each specific technique, its assumptions, evidence 
that these assumptions are met in each case of its use, and page 
references to use of the technique in the evaluation report (in- 
cluding summaries) should be cited. 



481 



c. The limits of findings based on statistical inference 
inherent in the particular data and the particular statistical 
techniques supporting them should be clearly and concisely stated. 
Confidence intervals and explicit description of the time, setting 
and participants to which the findings can be applied should be 
included . 

d. Ways in which the size or direction of program effective- 
ness evaluation results might be an artifact of statistical 
technique—for example, "statistical significance" — should be 
briefly indicated to give the legislative user an indication of 
possible practical weakness in the evaluation findings. Ideally, 
statistical analysis itself would not affect apparent results — 
but given complex situations, relatively small effects, and more 
than one right choice for the statistician to make, it is worth- 
while either explicitly to rule out such problems or to indicate 
their extent in an appendix. 

The second element of the strategy — independent, simultaneous 

statistical analyses of the data using alternative techniques — 

can keep the legislative user from becoming the captive of a frame 

of reference and assumptions which he may not accept. If the 

findings of alternative analyses of the data tend to coincide, 

corroborative evidence is provided; or, if they diverge it is 

perhaps the issues surrounding program effectiveness which should 

be debated and not the degree of effectiveness. Of course, all 

alternative statistical analyses would have to be responsibly 

conducted and documented as recommended above. Independent, 

simultaneous statistical analyses are relatively inexpensive compared 

to the cost of developing measurement instruments and collecting 

the raw data. In practical terms it would require that those who 

actually collect the raw data store it on computer tapes in such 

a way that they are readily usable by other professional evaluators 

to whom the raw data would be made available for alternative analyses 

In the past, alternative analyses of raw data have been made 



482 



sequentially rather than simultaneously with the result that their 
finding have not been available for corroboration of results or 
sharpening of Issues when needed by legislative users. And, such 
sequential analyses of data gathered at one point in time may take 
so long that the raw data are obsolete with respect to present or 
future situations with which the legislative user may be concerned. 
Any alternative statistical analyses should be made simultaneously . 

Statistical analysis is the last chance for bias or invalid 
evidence to be obscured in highly technical problems and language. 
The documentation and opportunity for alternative analyses suggested 
above create a situation which would minimize the possibility that 
such bias or invalid evidence would escape the attention of the 
legislative user. 

Step 7: Format for Evaluation Studies . 

Reports of evaluation studies should be written in a concise 
format which exclusively links the legislative goal(s) at hand to 
the program results. 

The format we have in mind would be more similar to a legal 
brief than to the format in which evaluation results typically 
have been reported. Cross-referencing of evidence to their data 
base is essential. 

Findings should be cross-referenced to the specific evidence — 
i.e., to the raw data, measures, research design, and statistical 
analysis on- which they are based. The purpose of this format is 
to present evidence concisely in such a way that its relevance to 



483 



the Issue and its credibility can be examined and reviewed by 
Consrress. Thus an evaluation report to Congress should,, comprise) >. ' 
these cross-referenced sections: 

1. the statement of the goals (cross-reference to results) 
as drafted by the committee, 

2. the evidence which the evaluator argues is pertinent, 
credible data on that issue, <f* * 

3. sections documenting the analyzed data, the analytic 
techniques (i.e., statistical tests, etc.) the raw data, the methods 
by which they were gotten, and other appropriate technical details 
on which the credibility of the evidence might be challenged. 

Procedures for using evidence in this format are described 

in Appendix A. 

Beware of Summaries : 

Although summaries are often suggested as a solution to 
Congress' information overload, they discourage andoften pre-empt 
Congress 1 rights and responsibilities to examine evidence directly 
and draw its own conclusions. Hence, we recommend against the 
summary as an exclusive vehicle for presenting evidence from evalu- 
ation studies to Congressmen and their staff. 
• ***»•*«* 

Why do this ? The practical purpose of the steps we have 
outlined here is to provide sufficient evidence to justify and 
document 

1. Committee actions at authorization stage 

2. Individual members' floor votes 

3. Actions of Appropriations and Budget Committees 

4. Challenge to Executive branch budgets. 



484 



The new Congressional budgeting system will require a flow 
of evidence at a relatively specific level and in readily usable 
and checkable form. Applying the various types of evaluation 
methods can provide that evidence if organized and applied as 
outlined here. 






485 

APPENDIX A 
AN ALTERNATIVE FOR EVALUATION REPORTING 

Congressmen can use evaluations to arrive at Congressional 
decisions or to review executive branch proposals — especially the 
Presidential budget and its documentation. The committee hearing 
can be a vehicle for transmitting and organizing the evidence 
provided by evaluation studies. The actual reports of evaluation 
studies can be presented in a concise, cross-referenced format, 
similar to simplified legal briefs. Principals and steps which can 
be taken to organize hearings to provide an efficient education in 
the facts at hand on any specific issue will be outlined below. 

Identification of issues . Hearings should be organized around 
a clear statement of the issue which is of legislative interest. 
For example, the Introduction of new legislation raises the issue 
of the nature and extent of needs addressed by this legislation. 
The appropriate evidence can be gathered by needs assessment and 
policy analysis — as indicated in Table 1. If program implementation 
or service delivery is the issue, evidence should be reported from 
process evaluations. Other examples may be inferred from Table 1. 
The important point is that the issue be defined to focus the 
hearing and appropriate type of evidence be available when needed. 

Present evidence for and against the issue . Much of the 
fogginess which frustrates users of evaluation reports is often 
due to efforts to avoid or conceal bias in the study. Of course 
reports should be objective not an exercise in propaganda or 
entirely personal opinion, but they will be most useful if organized 



486 



into clear cut documentation of positive or negative positions 
on the issue. For example, if the implementation of a program is 
at issue you need to have evidence for and against the points of 
implementation at issue. 

If evidence in testimony is systematically keyed to different 
sides of an issue, that evidence should be examined and used more 
readily than if it is varied along lengthy caveats. Advanced 
categorization of the evidence in relation to the issue can enhance 
Congressmen's responsibility to make a layman's judgement on the 
validity, relevance, and sufficiency of technical evidence for 
differing positions. 

Setting reporting and hearing dates. With a little thought 
you can identify advance broad issues which will arise, in connec- 
tion with new bills, existing acts, or the budgeting/appropriations 
process — outlined in Table 1 — and arrange to develop the kind 
of data you will need via the legislative language in Table 2 A in 
the congressional veto process. You must also specify reporting 
dates for receipt of evaluation briefs which will coincide with the 
times at which issues will be most salient. Most routine but 
significant issues can be foreseen accurately enough to be built 
into a committee schedule. 

"Cross-examination" questions and staff review . On receipt 
of an evaluation study brief, committee staff should review the 
brief and formulate a list of questions for circulation to members 
which bear on the relevance, validity and sufficiency of the evidence 
in the brief. These questions should be circulated to members in 



487 



advance of the hearings, along with each copy of each brief. 
Such questions should be directed to the witness testifying the 
actual hearing. 

The hearing . The procedure outlined so far is aimed at 

< providing Congressmen with access to direct evidence. The hearing 
should be a forum in which challenges to that evidence can be 
raised and settled. Hence, the hearing should be conducted with 
the support of examination of witnesses by expert consultants on 
behalf of the committee if necessary. 

Results of the hearings: Workable reports . The product of a 
hearing report should be evidence which has been sifted and sorted 
by the committee to guide floor voting, for authorization 
committee support, for the work of the appropriations, budget or 
tax committees. Such committee reports can be a data base for mark-up 
decisions, committee decisions on Congressional budget resolutions, 
as well as for challenging the Presidential budget's responsibility. 
Appeals: Floor Votes and Subsequent Committees . The action 

,. of a committee can in effect be appealed by reexamination of evi- 
dence , or by providing new evidence at points of subsequent action. 
For example, in appropriation committees, budget committee or 

I floor voting. 

Finally, it should be noted that the format for reporting 

i evaluation studies should suit the needs of the committee. In 
some cases, this can be done in a committee hearing context, other 

1 times in informal meetings. In any case the major consideration, 
assuming a valid evaluation has been completed, is that the results 



488 



be reported in a form that will be useful to committees and member:*, 
of congress in their legislative deliberations. 



489 



APPENDIX B 



SUMMARIES OF EVALUATION TECHNIQUES 
(Being Developed) 



490 



Preprogram measurements on the measuring instruments that are 
to be used to define program effects are usually needed. Yet it is 
common for a new program to be implemented immediately, before the 
measuring instruments have been developed or applied. Where program 
effectiveness is to be measured with novel measurement devices (and 
this is the usual situation) and where the program is a pilot program 
being instituted to inform Congress about the desirability of a 
potential National Program, the legislation might well specify that 
program implementation may be deferred until the preprogram measure- 
ment activity has been completed. Legislation might allow as much 
as one year for measurement development and the full scale prepro- 
gram measurement activity. (Where appropriate field-tested measures 
already are available, a six-month period for organizing the mea- 
surement staff and executing the preprogram measurement activity 
would usually be sufficient.) 

Untreated comparison groups are often needed to make sure that 
the oreproerram-postprogram differences are not due to other events, 
such as inflation, recession, the shutting-down of a major employer, 
unusual weather conditions, increasing familiarity with the mea- 
suring instruments, and the like. If the measuring instruments 
are such that other probable events (in addition to the new program) 
are likely to cause changes, then the legislation should consider 
specifying that one half or more of the measurement effort be 
devoted to making measurements on comparison groups not receiving 
the treatment. At present, most evaluation staffs do not feel 
they have this authority, and many evaluations lack this much 



491 



APPENDIX C 



CONSIDERATIONS IN DESIGNING EVALUATION 



492 



needed comparison base. (If the program is expensive, it is usually 
cheaper to increase the statistical dependability of the results by 
hnvinp; considerably more persons or other units in the comparison 
group than in the program group. The common practice of having the 
comparison group smaller than the program group is neither eco- 
nomically or statistically efficient.^ 

Uncontrolled variation in the treatment being studied was 
seldom a serious problem in the scientific laboratory settings in 
which the designs now applied to program effectiveness evaluation 
were developed. In such scientific settings it is taken for granted 
that the kind of treatment being evaluated is carefully and fully 
described, is sufficiently simple to so describe, and is in fact 
provided as described . The wise user of program evaluation can 
make none of these assumptions about the program treatment whose 
effectiveness is being evaluated. However if the legislative user 
has taken our advice in Steps -j and 4 regarding measurement of pro- 
gram delivery, confounding of program effectiveness by uncontrolled 
variation in program services can be controlled in one of the 
three following ways. 

1. Make no deliberate effort to control variation in program 
services at the time they occur, but to require that measures of 
program services be used as post hoc controls in the statistical 
analyses of program effectiveness data. This alternative is 
slightly better than nothing, but often falters on technical problems. 

2. A second alternative is to require that a program be certified 
as complying- with the measurable definitions of the program obtained 



493 



in previous steps. Consistent compliance of program services 
throughout the period of effectiveness evaluation should be docu- 
mented. In effect, this alternative provides that what is to be 
evaluated for effectiveness has in fact been delivered as lntended -- 
an easy assumption in a laboratory, but not in the real world of 
Federal programs. 

3. Planned variation in program services is a third — and 
probably best — way to assure that the delivery of program treatment 
does not confound evaluation of its effectiveness. In planned 
variation the levels and types of program services are deliberately 
and systematically varied in a logical scheme which provides the 
evaluator with maximum opportunity to detect program effectiveness 
and associate it with program services. Planned variation requires 
full cooperation and coordination of those administering program 
services with program evaluators, but the benefits and additional 
knowledge gained thereby easily offset the extra effort required. 
Measurement of program services is a necessary condition for planned 
variation, as for the other alternatives. Thus we strongly recommend 
that legislative users of program evaluations insist that evalua- 
tions of program effectiveness include measurable definitions of 
program services, valid measurements for those services, and the 
best possible opportunity for evaluators to rule out uncontrolled 
variation in program treatments that might confound program effect- 
iveness . 



494 



Randomized selection procedures for admission to the program 
are often desirable to increase the comparability of the comparison 
croup. For example, in compensatory evaluation programs, it 
frequently turns out that the only schools that will volunteer for 
the tedious comparison measurements are high morale schools that 
have nothing to fear from having their educational success measured. 
Such comparison schools tend to make the needy program schools 
look bad even when the program is effective. Legislation might 
well permit a procedure whereby a lottery was used to select program 
schools and comparison schools from among applicant schools that 
had agreed to cooperate no matter which group the lottery assigned 
them to. If the program is a pilot program, or even an operational 
program that is underfunded, the use of randomization procedures 
to select program and comparison groups is a fair and democratic 
way of allocating scarce resources, and in addition makes program 
e valuations optimally interpretable. 

Randomization of program participation among eligible parti- 
cipants is desirable when, as is usual, the evaluator can neither 
know nor control all the characteristics of participants which could 
influence program effectiveness or the outcome measures. The 
principle of randomization provides procedures whereby every po- 
tential participant has an equal chance of receiving program 
services. Opportunity for randomization will permit use of much 
stronger evaluation designs by the evaluator, and thereby provide 
legislators with much better evidence for oversight. In authorizing 
use of randomization of program participation among those eligible, 



495 



legislative users of evaluation should of course be watchful to see 
that basic rights of participants are not compromised. Usually, 
however, randomization procedures serve to increase equality of 
opportunity rather than to restrict freedom. Coerced participation 
in a program is undesirable with or without randomization. 

Evaluation research design is extremely complex and requires 
many technical Judgments which cannot be made responsibly by the 
layman untrained in research design. Legislators, however, do not 
meet their own responsibility by merely surrendering the whole 
matter to experts. Legislators have a responsibility to raise 
basic questions about the validity of the evidence they will be 
getting in the program effectiveness evaluations they authorize 
for use in oversight. 

Quantified Eligibility criteria for programs , if strictly 
applied, make possible a second-best substitute for randomized 
selection. Most programs that are intended for the neediest (or 
the most deserving) are in practice administered in ways that 
preclude any interpretable comparison between the outcomes of those 
who get the program and those who do not. If instead, the admission 
criteria can be specified quantitatively (as in terms of income 
level, or reading comprehension scores, or multiple rating scales 
used by admission staffs, or explicit composites of such ingredients) 
if the program is then given to those scoring most eligible, if 
outcome measures are made not only for the program recipients but 
also for unaccepted applicants beyond the cutting point and if the 
outcome measures are analyzed in terms of the preprogram eligibility 



496 



scores, then a very useful estimate of program effectiveness can 
be made. To make this possible, legislation might well specify that 
the program be given to the most eligible applicants and that 
eligibility should be in terms of a quantitative index based upon 
specified factors. Legislation specifying a quantitative eligi- 
bility dimension but not a specific cut-off score is most useful. 
Especially when individuals are being selected (rather than schools, 
etc.) a specific cut-off score, if generally known, can lead po- 
tential applicants to fail to apply or to falsify their status 
(earnings, etc.), both effects tending to invalidate the analysis. 

Administrative records are often highly relevant to program 
evaluation. There are available ways of using them that fully 
protect the privacy of the individuals involved. It will often be 
optimal for legislation to specifically authorize s"uch appropriately 
safeguarded uses. It will rarely be desirable for the legislation to 
preclude such use. Many programs have among their goals modifying 
factors that are already being recorded by some government agency; 
earnings of the poor and undereducated , claims on unemployment 
insurance, school achievement scores. For such programs admini- 
strative records for program groups and comparison groups have 
many advantages. They can be much less expensive than to obtain 
information on the same topic through interviews. They can cross- 
validate interview findings. They avoid the differential cooperation 
and attrition often found between experimental and comparison groups. 
They allow for time-series, that is, for records on program and 
comparison groups for a number of years prior to the program and a 
number of years afterwards. 



497 



Such time series are often convincing evidence of program 
effectiveness even without a comparison group. With a comparison 
group, they make for the strongest evaluation design possible. 

Competitive replication with variation is an extremely impor- 
tant source of validity in science which can easily be lost in 
large-scale program evaluation. Often It would be better to have 
the program evaluated by numerous smaller scale independent evalu- 
ations than by one large-scale effort. At^ very least, legislators 
should consider splitting all major evaluations into two, each to be 
implemented by a separate, independent research team. The measure- 
ment procedures would have the same goals but different details. 
The many arbitrary decisions that have to be made in any research 
effort would be made differently. If two such independent evalu- 
ations were to come up with the same recommendations, the policy 
implications would be clear. If they were to disagree, Congress 
would be thereby informed as to the equivocality of the evaluations. 

Competitive reanalysis of program evaluation data is another 
aid to objectivity which legislators can specifically encourage, 
by requiring that all data from all measures collected be made 
available for reanalysis by other persons (subject to deletion of 
personal identifiers and other protection against disclosure of 
information on individuals). This issue is discussed in more detail 
in the manual. 



498 





























— — 
• 








• « 


> • 


























• 


• 


• 













o 


o 


o 










O 


O 








o 


o 





o 






























• 


• 


• 


































• 


• 








• 


• 






















• 










• 


• 




































• 






























• 


• 


















• 


• 














• 


• 






































• 


• 




• 










• 














• 


• 


• 


• 












































• 


• 








• 






• 




• 


• 


• 


• 


• 


• 










• 








o 


o 





o 


o 


o 


o 


6 


o 


o 


o 


o 






o 


o 


o 


o c 


o 


?! 

-j. 

ii 

•t\ 

La 

I 

1 


i 


si 

\\ 

I 


I 

i 

i 

! 

•s 
I 

f 

i 

I 


? 

5 

r 

? 

r 
i 


11 
I ? 

fl 

i 

r 


is 

is 
s-s 

° ? 
i 


| 


1st 

&5| 
s « -T 

III 

Hi 
Iff 
III 


is 

II 

i 
l 

! 


ill 
■ji 

til 
III 

sji 
•1 ■ 

5 ! V 

!l! 

pi 


11 

is 
Is 

1! 

|i 

! 


1:1! 

1*1! 
[ill 

sell 

si *s- 

. ?•:" 

lira 

Cs Eg 

UH 
'If 


"I s 
its 

,r e 

8.7 

It 
\\ 

3' 

:z 

I 


-1 
•I 


sl'l 

L|-f| 

iS J 

iti 

s -i 

-•8 -, 
5 1° 

III 


If 
s - ! 

l 


II ! 

1 - 
s 

| : 
: 1 

: i 
s- s 

- i 
i s 


1 

? 
I 


t 


i 

i 


1 


I 

s 
1 

: 
1 

i 

£ 


I 


f 

1 

i 

1 

1 

! 


i 

jf 
J 

| 


1 
I 

1 

t 

I 


! 

; 
§ 

i ' 
I 

; 

t 


ff] 

s : 

V 

If 

H 

1 • 

fl 
it 

f' 


j 

5 
| 

1 

| 

j 

! 


1 

1 

£ 
1 

1 


i 

i 

8 
5 
1 

i 
I 

I 
s 

I 
i 

s 
| 
s 
| 


1 
jj 

1 


j 

| 
5 

j 

s 


i 

I 

I 

1 

I 


J i 
1 

fl 

«; 

IS 

l \ 
§> 

5 


i ] 

j ] 

«" ! 
1 j 

I | 

! j 

i j 

f 1 
? | 

1 


II 
1 

1 

1 

I 

s 

1 

» 

i 

r 

1 



*^ 






111 

8*3 
SIS 

|| 



HI 

Sli 



its 

is I 

lit 

■ il 



i 

I 

i 

a 

U 

6 



APPENDIX E 

75-15 SP 



PROGRAM EVALUATION: EMERGING ISSUES OF 
POSSIBLE LEGISLATIVE CONCERN RELATING TO 



THE CONDUCT AND USE OF EVALUATION IN THE 
CONGRESS AND THE EXECUTIVE BRANCH 




500 



TABLE OF CONTENTS 



ABSTRACT 

PART I . INTRODUCTION 1 

PART II . THE INITIATION AND USE OF PROGRAM EVALUATION 7 

A. The Need to Clarify Legislative Statements of Program Goals 

and the Purposes of Evaluation , 9 

1. Discrepancies between the Purposes of Evaluation and the 
Techniques of Evaluation 9 

2. Recommendations to Specify Program Goals in 

Legislation 11 

3. Criticisms of Specifying Program Goals in 

Legislation 13 

B. Better Communication for Effective Evaluation 14 

C. The Integration of Program Evaluation into Program 

Management 17 

1. Incorporation of Evaluation into Agency Program 

Planning and Budgeting 17 

2. The Development of Evaluation Work Plans 19 

3. LEAA: An Illustration of These Deficiencies 19 

4. The Contribution of Evaluation Work Plans to 

Evaluations Performed at the State Level 20 

5. Proposals for Comprehensive Evaluation Plans 21 

6. Proposals for a "Preassessment of the Evaluability of 
Programs" 22 

D. The Need for Better Dissemination of information on Program 
Evaluations Supported Under Federal Agency Auspices 23 

1. Investigation by the General Accounting Office 24 

2. NIMH/UCLA Project on Data Bank of Program 

Evaluations (DOPE) 24 

PART III. IMPROVED COORDINATION OF PROGRAM EVALUATION IN THE 

EXECUTIVE AND LEGISLATIVE BRANCHES 26 

A. Issues Surrounding the Need to Incorporate Program 

Evaluation Findings into Executive Branch Budget Processes.... 26 

B. Proposals to Give the Office of Management and Budget 

Program Evaluation Responsibilities 29 

C. The Proposal to Establish a Federal Program Evaluation Office 
in Either the Office of Management and Budget or in the 
Executive Office of the President 29 

D. Current 0MB Initiatives in Program Evaluation 30 

E. Issues Relating to the Improvement of Legislative 

Branch Program Evaluation Capabilities 32 

F. Issues Surrounding the Role of the General Accounting Office 
in Providing the Congress with Improved Program Evaluation 
Capability 36 

1. Basis of GAO's Statutory Authority for Program 

Evaluation 36 

2. Evolution of GAO's Program Evaluation Activities 36 



501 

TABLE OF CONTENTS (cont.) 

Page 

3. Illustrations Provided by the GAO of Its 

Program Activities 38 

4. Other GAO Program Evaluation Activities 39 

5. Activities and Legislation in Support of Giving GAO 

More Responsibility for Program Evaluation 39 

PART IV. THE PROCUREMENT AND CONDUCT OF PROGRAM EVALUATION RESEARCH... 41 

A. Issues Relating to the Funding of Program Evaluation 

Research 42 

1. The Lack of Appropriate Information 42 

B. Difficulties of Obtaining Information on Types of 

Performers of Program Evaluation Research 45 

C. Specific Issues Relating to Federally Funded Research 
and Development Centers as Performers of Program 

Evaluation Research 47 

D. Issues Relating to Funding Mechanisms (Grants and 

Contracts) in Program Evaluation Research 49 

1. Issues Relating to Profit- Versus Non-Prof it-Making 

Performers 49 

E. Legislative Requirements to Mandate a Certain Percentage 

of Funding for Program Evaluation 50 

F. Alternative Institutions for the Conduct of Program Evaluation 
Research Under Federal Agency Auspices 51 

PART V. METHODOLOGICAL ISSUES IN THE CONDUCT OF PROGRAM EVALUATION 58 

A. Experimental and Non-experimental Research Designs 58 

B. Scope and Limits of Social Experimentation 59 

1. Legislative Interest in the Ethics of 

Experimentation 65 

C. The Need for Continuous, Appropriate Impact Data 65 

D. Cross-fertilization Between Advances in Social Indicators 
Research and Improvements of Data Required for Program 
Evaluation 69 

PART VI. THE POSSIBLE ISSUE8 OF LEGISLATIVE CONCERN RAISED BY 

INFORMATION PRESENTED IN THE REPORT 75 



502 



The conduct of program evaluation research has grown considerably In recent years, 
in response to explicit legislative mandate as well as executive agency directive. How- 
ever criticism abounds about the utility to the government of many of these types of re- 
search. This report discusses some of the factors both governmental and non-govern- 
mental which contribute to this situation. It also identifies recent actions taken to 
remedy the discrepancy between the promises and utility of evaluation research. Re- 
maining issues of possible legislative concern are also listed. 



303 



PART I. INTRODUCTION 

Federal expenditures for program evaluation are estimated to have risen by more 
Kan 500 percent from 1969 to 1974, from $20 million to more than $130 million. This 
ncrease in program evaluation activities reflects two Important trends. The first 
.s a shift in total Federal budget expenditures toward greater emphasis on human re- 
sources programs, from approximately $65 billion in 1969 to almost $173 billion in 
1976. _1/ The second trend consists of a "many-fold multiplication" in congressional 
lppropriations for evaluations in response to increasing demands from the Congress, 
:he President, and executive agencies for greater accountability and evaluation of 
che resources devoted to social programs. _2/ The conduct of program evaluation is 
also part of a broader trend to provide decisionmakers in both executive and legisla- 
tive branches of Government with objective information or policy advice on the effects 
of Federal actions and programs, preferably before large-scale application. _j/ Dr. 



_1/ Executive Office of the President. The United States Budget in Brief, Fiscal 
Year 1976, Washington, D.C., U.S. Govt. Print. Off., 1973. p. 7. 

]J Mushkin, Selma J. Evaluation: Use With Caution. Evaluation, v. 1, 1973! 31. 
The Senate Agriculture Committee's needs for evaluation especially an evaluation 
of the Farmer's Home Administration are described in James W. Giltmier. Policy 
Formation Through Program Evaluation and Systems Analysis; A Congressional View, 
mimeo, 20 pages, paper presented before Evaluation Seminar, (held monthly under 
Sen. Brock's authority), Seminar held November 20, 1974, U.S. Capitol. 

2_l Program evaluation activities required under the Congressional Budget and Impound- 
ment Control Act are described below in this Introduction, In addition, The Office 
of Technology Assessment, established under P.L, 92-484, October 13, 1972, as an 
information and analysis office attached to the Congress, will "...provide [the 
Congress with] early indications of the probable beneficial and adverse impacts 
of the applications of technology..." and other relevant information, to assist 
in the legislative process. [See: U.S. Congress. Senate. Committee on Rules 
and Administration. Subcommittee on Computer Services, Technology Assessment 
for the Congress, staff study, November 1, 1972, 92nd Congress, 2nd session, 
Washington, D.C. U.S. Government Printing Office, 1972. (Committee Print,)] 
Judicial Interpretation of the National Environmental Policy Act of 1969, now 
reflected In revised guidelines issued by the Council on Environmental Quality, 
requires the submission of 102 impact statements on a wide variety of major Fed- 
eral activities which may impact on the "human environment," broadly defined. 
These activities Include construction of facilities, alterations to the natural 
physical environment, funding proposals, and administrative actions, as well as 
proposals for different types of. Federally supported scientific research and 



U-867 O - 76 - 33 



timated 



504 



H. Guyford Stever, Che President's Science Adviser estimates that $835 million will 
be obligated for social R and D programs in the fiscal year 1975. In fact in dis- 
cussing the 1975 budget message he singled out for special consideration as social ex 
periments designed to generate information for public policymaking, three programs in 
health insurance, income maintenance, and housing allowances, with obligations es 
to total $33.9 million. _lj 

Concurrent with increasing activity in social program evaluation, however, there 
is evidence of heightened concern by Members of Congress, agency program administra- 
tors, and social scientists, with inadequacies in program evaluation methodology, pro- 
cedures for data collection, procurement, administration, and utility of program eval- 
uation, especially in planning, oversight, and budgeting functions. 

The history of Federal "social program evaluation" activities has never been fully 
chronicled. Similarly, there is no detailed analysis of the merits of evaluation ob- 
jectives, programs, and approaches used by different social program-oriented agencies. 
This report is designed to provide the Congress with background information on existing 
and emerging issues of concern relating to evaluation. Part I is this introduction. 
Part II deals with some general issues in program evaluation: its objectives, commu- 
nications between researchers and program managers , and the difficulties of using eval- 
uation in agency management procedures. Part III treats 0MB program evaluation activ- 
ities, proposals for interagency evaluation mechanisms, and activities surrounding im- 
provement of legislative branch programs evaluation capabilities. Part IV deals with 
issues relating to the procurement and conduct of program evaluation research from the 
point of view of both the evaluation researcher and the government user. Part V deals 
with methodological issues in the conduct of program evaluation. The possible issues 



development which may result in development of technology with a potential impact 
on the environment. (See: Council on Environmental Quality. Preparation of 
Environmental Impact Statements, Guidelines. Federal Register, v. 38, no. 14, 
Part II, August 1, 1973). 
_4/ The Science Adviser, "Federal Budget for Science and Technology - FY 1975," 
February 4, 1974: 8. 



505 



CRS - 3 

it legislative concern raised by information presented in the report are collated in 
J art VI. 

The term "program evaluation" is used in two general ways in the Federal governnent. 
k.t a very aggregated level, the term denotes a process of assessing cost and effective- 
ness to ascertain the appropriate funding level for a program in relation to other 
aational goals. At a less aggregated level, the term refers to the processes of eval- 
uating a particular agency's program to determine whether and to what extent it achieves 
its purposes (impact evaluation), and whether it is administered efficiently (process 
evaluation). Decisionmakers use these evaluations in deciding whether a program should 
be continued, modified, terminated, or replaced by another "more beneficial and more 
effective" program. 

The term "program evaluation" has been associated with any activity to measure the 
effectiveness and outcome of a program. However, it has been most widely associated in 
recent years with evaluation of major social programs, such as educational intervention 
strategies, work incentives, and juvenile training. In this respect, evaluation, ac- 
cording to Wholey, et. al., provides "objective information to program managers and 
policy-makers on the costs and effects of national [social] programs and local projects, 
thereby assisting in [promoting] effective management and efficient allocation of lim- 
ited resources." V 

Federal agencies have developed a number of different types of program evaluation 
mechanisms. The Congress mandates some evaluations, which are conducted at stipulated 
funding levels and with mandated techniques. Some agencies initiate evaluation to pro- 
vide information for program planning and budget justifications. Effective coordinated 
program evaluation offices exist in some agencies. Other agencies have not organized 
their program evaluation activities formally. 



_5/ Wholey, Joseph S., et. al. Federal Evaluation Policy: Analyzing the Effects of 
Public Programs. Washington, D.C., The Urban Institute [1970] pp. 19-20. 



506 



One characteristic is common to almost all social program evaluations that is 

methodology. Thus according to Wholey, et. al. , "evaluation is research, the applica- 
tion of the scientific method ... to learn what happens as a result of program activi- 
ties." Evaluation thus includes: 

the definition of program objectives, the development of measures of progress 
toward these objectives, the assessment of what difference public programs 
actually make, and the projection of what reasonably could be expected if the 
program were continued or expanded. _6/ 

The tools and techniques of program evaluation usually incorporate the basic techniques 

of social science research systematic collection of evidence from a representative 

sample of the population; translation of information into quantitative terms; and anal- 
ysis of the information, using statistical manipulation and reference to theoretical 
notions about human behavior, to "draw conclusions about the effectiveness, the merit, 
the success, of the phenomenon under study." 7 / 

Much of the current criticisms of program evaluation research relate to its state- 
of-the-art and its questionable utility to the government. A number of recent or on- 
going governmental and professional activities illustrate these concerns. 

The need to improve the state-of-the-art of program evaluation is underscored by 
the establishment within the Research Applied to National Needs program (RANN) , of the 
National Science Foundation, of a $2.5 million program to provide "rigorous examination" 
and refinement of the techniques of program evaluation and policy-oriented research in 
the fields of labor, youth, health, and criminal justice. 8 / 

The Science and Technology Policy Office of the National Science Foundation is sup- 
porting three research projects on social science policy, designed in part to improve 
the conduct and use of program evaluation. These studies are: 



J_l Weiss, Carol H. Evaluation Research: Methods for Assessing Program Effectiveness. 
Englewood Cliffs, Prentice-Hall [1972] p. 2. 

8 / "NSF Initiates Programs for Evaluation of Policy-Related Research Concerned with 
Human Resources." NSF News Release, April 11, 1973. (NSF 73-137.) 



507 



National Academy of Sciences, Study of Social Research and Development and its 
Role in Policy Making, 6 mos., $116,500; 

Social Science Research Council, Conference on Social Experimentation, 12 mos., 
$23,100; and 

University of Washington, Social Policy Analysis and Research in Social Policy- 
making, 14 mos., $50,000. 

Some of these topics were addressed during two conferences held at Dartmouth during 
the summer of 1974. The first, a conference on Social Experimentation, was sponsored by 
the Social Science Research Council with funding from the National Science Foundation. 
The second, the "Dartmouth/Organization for Economic Co-Operation and Development (OECD) , 
Seminar on Social Research and Public Policies," was supported with funds from the Ford 
Foundation, Dartmouth, and the OECD. The Fifth Annual Conference of the National Capi- 
tal Area Chapter of the American Society for Public Administration had a session devoted 
to examining the conduct and utility of program evaluation. It was h'eld on November 14 
and 15, 1974 in Washington, D.C. 

Numerous bills were introduced during the 93rd Congress to incorporate rigorous 
program evaluation, pilot test, and cost-benefit analysis procedures into legislation 
in order to improve congressional oversight of the budget. The Congressional Budget and 
Impoundment Control Act of 1974 (P.L. 93-344, approved July 12, 1974), established a 
Program Evaluation Office in the General Accounting Office and gives the agency added 
responsibility for program evaluation. It also strengthens legislative responsibility 
for using program evaluation information. 

The topic of assessing the quality of evaluation research, especially experimenta- 
tion, for decisionmaking processes is also receiving some attention by the Special Com- 
mittee on Federal Agency Evaluation Research of the Division of Behavioral Sciences of 
the National Research Council. The Office of Management and Budget has also established 
an Office of Evaluation and Program Implementation to assess these issues. In addition, 
the General Accounting Office has published several reports directed to congressional 
committee chairmen, recommending steps to be taken to refine legislative language 



508 



requiring program evaluation. The GAO is now preparing a guide for program evaluation 
standards, expected to be released in the Spring of 1975. 

An effort is made in this report to present background information in a manner 
which serves the interests of the Congress. Many of the issues raised may be relevant 
to both House and Senate Budget Committees and the Congressional Budget Office, estab- 
lished under the Congressional Budget and Impoundment Control Act of 1974, P.L. 93-344 
of July 12, 1974. In addition some authorizing committees as well as committees re- 
sponsible more generally for overseeing the organization and administration of Govern- 
ment might possibly be concerned with the issues raised. 

The following format is used in presenting the material. Each issue is developed 
by referring to current activities and literature, such as those of the Office of Manage- 
ment and Budget, the General Accounting Office, the National Science Foundation, and the 
National Academy of Sciences; social science critiques; and published" and unpublished 
Federal agency reports. Also drawn upon if applicable, are legislative activities which 
address these topics. If available, recommendations for improvement from published 
sources are given. The possible issues of legislative concern generated by the report 
are presented in Part VI. 

(It should be noted that the Library Services Division of the Congressional Research 
Service has published multilithed annotated bibliographies on program evaluation. All 
were prepared by Nancy Davenport. They a:e: Evaluation Research in Public Administra- 
tion: Selected References, 1967-1974, October 21, 1974,. 9 p.; Evaluations of Manpower 
Training Programs: Selected References, 1970-1974, October 25, 1974, 8 p.; and Evalua- 
tion Research in Social Policy: Selected References, 1970-1974, October 31, 1974, 8 p.) 



509 



PART II. THE INITIATION AND USE OF PROGRAM EVALUATION 

The General Accounting Office (GAO) , estimates that "Federal expenditures for non- 
defense program evaluation have risen dramatically in the past few years from less 

than $20 million in the fiscal year 1969 to at least $110 million in fiscal year 1972 _9_/ 
and $130 million in the fiscal year 1974. 10/ As a result, according to the GAO: 

In response [to the increasing demand for program evaluation] executive agencies 
have set up new evaluation offices and expanded existing offices. Contracting 
for evaluation studies has expanded greatly. The trend in number and size of 
evaluation studies sponsored by the Federal Government has been upward. Private 
foundations as well have stepped up funding of evaluation studies. State and 
local governments have increased evaluation efforts and several universities have 
set up evaluation institutions. 11/ 

Although program evaluation is extensively used, there is widespread criticism of 
its utility to decision-makers. For instance, Sen. William V. Roth, Jr., reporting to 
the Congress on a 1972 staff study entitled R eport on Survey of Federal Program Evalua- 
tion Practices , said "the study conducted by my staff suggests serious weaknesses in 
agency evaluative and analytical procedures." The report, he continued, "call[s] at- 
tention to the need for the executive branch to improve and extend its attempts to 



9 / U.S. General Accounting Office. Program Evaluation: Legislative Language and a 
User's Guide to Selected Sources. Washington, D.C., U.S. General Accounting Of- 
fice [June 1973] p. 1. It should be noted that most of the literature on program 
evaluation indicates increasing demand by both Federal agencies and the Congress, 
and increased funding by agencies to support the conduct of program evaluations. 
Dr. Allen Schick disagrees. He reports in a 1973 publication: "...There has been 
a definite drop of interest in large program evaluation, though many Federal agencies 
are trying to institutionalize evaluation through the collection of output and per- 
formance data." (Allen Schick (Senior Specialist, Congressional Research Service, 
Library of Congress) ,• "The Pilot Testing of New Programs: An Analysis of Title IV, 
S. 3984 (92nd Congress)," In U.S. Congress. Joint Economic Committee. Subcom- 
mittee on Budgeting, Management, and Expenditures. Improving Congressional Con- 
trol Over the Budget: A Compendium of Materials, March 27, 1973. Washington, 
D.C., U.S. Govt. Print. Off., 1973. p. 276. (Committee Print.) 

10/ "Program Evaluation in the Executive Branch: Estimate of Funds Budgeted for Fis- 
cal 1974," in U.S. Congress. Joint Committee on Congressional Operations. Con- 
gressional Research Support and Information Services: A Compendium of Materials. 
Washington, U.S. Govt. Print. Off. [May 1974] pp. 289-290. 

1] 7 Program Evaluation: Legislative Language and a User's Guide to Selected Sources, 
op. cit.: p. 1. 



510 



measure the accomplishments of governmental activities and weigh these accomplishments 
against their costs." 12/ Federal administrators who fund and oversee program evalua- 
tion research have also criticized its utility. For instance, Robert G. Bruce, Assistant 
Administrator for Program Planning and Evaluation, Social and Rehabilitation Service, 
HEW, reports: "We might as well be candid: Federal program evaluations so far have 
been largely ineffective." 12/ Garth Buchanan and Joseph Wholey, program evaluation 
researchers at the Urban Institute, a research and evaluation agency created first to 
serve the Department of Housing and Urban Development, observe similarly: 

. . . While we have seen that increasing amounts of money are being budgeted for 
evaluation, we conclude that this increasing support must be due to a continuing 
recognition by the Government thai evaluation information is needed for the devel- 
opment and management of social programs, rather than to the recognition that 
evaluation as currently practiced is the answer to these needs. We are led to 
this conclusion because in our judgment, the impact of evaluation results on pro- 
gram development and improvement over the last two years has been disappointing 
when compared with the amount of money and effort that has gone into evaluation, j^/ 

The Office of Management and Budget recently initiated a major effort to improve 

Federal agency program evaluation capabilities. Agency spokesmen identify four basic 

difficulties with current Federal program evaluation research. Their concerns parallel 

those raised above. They are: (1) timeliness for decisionmaking; (2) reliability of 

the data used and the methods employed; (3) relevance of evaluation activities to major 

policymaking needs; and (4) insuring that program evaluation reports are made available 

to and utilized by decisionmakers. _15/ 



12/ Public Program Analysis and Evaluation for the Purposes of the Executive and the 

Congress. Statement of the Hon. William V. Roth on the Floor of the Senate. Con- 
gressional Record, June 8, 1972 (daily edition): S. 9026. The study referred to 
is: Report on Survey of Federal Program Evaluation Practices Conducted by the Staff 
of Senator William V. Roth, Jr., (Delaware), May 1972 : 86 pp. (Available from 
Senator Roth's office.) 

!3/ Bruce, Robert G. What Goes Wrong with Evaluation and How to Prevent It. Human 
Needs. (Publication of Office of the Administrator, Social and Rehabilitation 
Service, Department of Health, Education and Welfare.) v. I, no. 1, 1972. 

!4 / Buchanan, v Garth N. and Joseph S. Wholey. Federal Level Evaluation, Evaluation Fall 
1972: 21-22. Weiss holds similarly that much evaluation research falls short of 
the expectations of funders, program staff and evaluators. See her current arti- 
cle: Weiss, Carol H. Between the Cup and the Lip ... Evaluation, v. I., no. 1, 
1973: 49-55. 

15 / Lewis, Frank L. and Frank G. Zarb. Federal Program Evaluation From the 0MB Per- 
spective. Public Administration Review, n. 4, July/August 1974: 314. 



511 



The literature on improving the administration of program evaluation raises im- 
portant issues which may warrant additional legislative concern. These, which will be 
, developed below, are to: clarify the purposes of evaluation; enhance communication be- 
tween administrators and program evaluators; integrate program evaluation into program 
management; and establish better information dissemination systems. 

A. The Need to Clarify Legislative Statements of Program Goals and the Purposes of 
Evaluation . 

According to the literature, much of the failure of program evaluation can be at- 
tributed to policymakers and legislators who write vague statements of program goals, 
which give evaluators imprecise yardsticks or standards against which to measure pro- 
gram accomplishments and deficiencies or costs and benefits. Dr. Bruce succinctly char- 
acterized this lack of definition: 

Program goals have to be carefully defined. There must be clear statements of 
what a program is supposed to accomplish — clear in the sense that you can pick 
them up and go out and measure whether or not you have achieved the program goals. 
Too often there is a tendency to rely upon broad statements of legislative intent 
to describe the purpose of a program. Such statements only provide a context for 
defining specific and measurable program outcomes. A lack of such agreed upon 
definitions to serve as criteria for evaluating a program's performance may be 
the largest single cause of ineffective evaluation studies. 16 / 

1. Discrepancies Between the Purposes of Evaluation and the Techniques of Evalua- 
tion . The literature reflects several variants on this theme of the need to specify 
program objectives and legislative requirements for program evaluation. 

One these receiving attention is: many programs are not amenable to evaluation be- 
cause of the lack of correspondence between their broad objectives and the tools and 
, techniques of evaluation research. On this point, Thomas Morehouse reports that Federal 
social services programs take one of two forms: maintenance programs or opportunity 
programs. A maintenance program "provides tangible goods or services to a clearly de- 
fined population, such as food or money. An opportunity program, in contrast, does not 
directly meet the material needs of a population, but is designed to "increase the group's 

}^J Bruce, op. cit. 



512 



capabilities or opportunities to acquire goods and services (and sometimes status and 
power) for themselves." These opportunities include providing the individual with job 
training or education to give him the skills needed to obtain directly goods and ser- 
vices. Innovative social programs, «,n which most program evaluations are required, 
are designed to provide opportunity, not goods or services. "The consequence for 
evaluation," Morehouse reports, "is that program 'outputs' are often more elusive and 
less easily measured than the outputs of tangible maintenance benefit programs." In- 
stead, these programs are "explorations of problems, objectives, and means." Evaluators 
therefore are forced to evaluate program performance by using input measures, such as 
manpower allocated to administer the program or funds spent. Such techniques yield very 
little information about the effects of the intervention strategy underlying the pro- 
gram. Morehouse concludes that social scientists who conduct evaluations are criticized 
by policymakers and administrators who "continue to call for 'objective information' 
and demand 'rigorous scientific' evaluation," and by fellow social scientists who call 
program evaluation research trivia. 17/ 

A second widely addressed theme is that non-specific or non-quantifiable state- 
ments of legislative program objectives militate against evaluation because broad-ganged 
programs cannot be evaluated easily with existing social science evaluation techniques 
which are oriented toward experimentation. 18 / The experimental approach, Morehouse 
notes, requires that the following criteria be met: "it assumes that a program has well- 
defined objectives and that its effectiveness can be determined by measuring the extent 
to which objectives are achieved; the setting in which the program operates should be 
reasonably "controlled"; the program "treatment should be reasonably uniform; and the 



17 / He adds that this is true "...even if performance (using experimentation and 

quantitative techniques) has often fallen far short of aspirations, and even if 
many evaluation research projects have been carried out in other, less highly re- 
garded ways." (Morehouse, 871.) The basic source is Thomas A. Morehouse. Program 
Evaluation: Social Research Versus Public Policy. Public Administration Review 
[November/December 1972]: 872. 

/ For "a explanation of social experimentation and its use in social irop-ii evalua- 
tion research, see Part V of this report. 



513 



program treatment should be applied In a large enough number of cases to provide an ade- 
quate sample of program experience." 19/ However, "most of the Federal social action 
and economic development programs to which evaluation requirements have been attached 
do not fit this mold." He continues: 

[They often have] broad aims that cannot be specified in clear-cut form; they may 
be concerned more with changing a situation than with affecting a large number 
of individuals directly; they generally operate in open, uncontrolled community 
and regional settings; and a given program tends to take different forms in dif- 
ferent situations. The programs, in short, are not themselves designed or ad- 
ministered as experiments. Thus it is rarely that even the basic data collection 
requirements for evaluation research are structured into program operations, let 
alone the requirements for standardized treatment, control-comparison, and repre- 
sentative samples. 20/ 

2. Recommendations to Specify Program Goals in Legislation . Most evaluation in 
recent years constitutes a response to legislative requirements incorporated implicitly 
or explicitly into authorizing legislation. Four recent reports have criticized the 
vagueness of these legislative requirements and offered suggestions for making them more 
specific, to conform with the methods and procedures of current evaluation practices and 
to the resources authorized to conduct evaluations . 21/ 

For example, in March of 1974, the General Accounting Office provided the Congress 
with an inventory and analysis of legislative language requiring program evaluation. 
This report, and an earlier one submitted in August of 1972, were designed to provide 
GAO with information to carry out its expanded evaluation responsibilities under the 
Legislative Reorganization Act of 1970. 22 / They were intended also to assist congres- 
sional committees in carrying out legislative analysis functions required under section 



19/ Idem 

20/ Ibid., pp. 871-872. 

'21 / GAO reports cited below; Wholey, et. al. Federal Evaluation Policy, op. cit.: 57- 
58; and National Advisory Council on Education Professions Development. Search 
For Success: Toward Policy on Educational Evaluation. A Report of the National 
Advisory Council on Education Professions Development, Washington, U.S. Govt. Print. 
Off., June 1974. pp. 35-36. 

22/ Legislative References to Evaluation, 1967 (62 pages); preceded by a Letter from 

the Comptroller General to the Chairmen of each Congressional Committee, August 11, 
1972. (GAO No. B-161740) ; and "Updated Part II of Program Evaluation: Legisla- 
tive Language," in Congressional Research Support and Information Services, op. 
cit., pp. 338-339. 



514 



136 of the Legislative Reorganization Act of 19A6, amended in 1970. 23/ The General 
Accounting Office surveyed legislative requirements for program evaluation assessing 
their wording along several dimensions: official responsible for evaluation, measures 
of effectiveness, methods required, report recipients, report dates, nature of recom- 
mendations, and funds authorized. The following conclusions were derived: 

(1) Of the 40 acts including evaluation references: 
15 acts are in the health and safety area; 

5 acts are in the education area; 
4 acts are in the transportation area; 
3 acts are in the law enforcement area; 
2 acts are in the housing area; 
2 acts are in the environmental area; 
2 acts are in the international area; 
2 acts are in the agricultural area; 
2 acts are in the economic opportunity area; 

the remaining acts are each in a different area-aging, selective 
service, and employment. 

(2) Multiple references to evaluation (more than one section on evaluation) 
are included in 19 of the 40 acts. 

(3) Specific authorization for funding evaluation is included in only 6 of 
the 40 acts. 

(4) Specific wording concerning measures of effectiveness is included in 24 
acts. 

(5) Eight acts include language specifying methods of data collection and 
analysis. However, each title of the Economic Opportunity Act includes very 
specific and detailed language on methods of evaluation including cost-benefit 
analysis, use of control groups, and standards for evaluation. 

(6) Specific reporting date(s) for evaluation results are included in 36 
acts. Reporting requirements range from a very general time span (annually) to 
a very specific date (on or before January 31, for example) . 

(7) There is a wide range concerning who is specified as responsible for con- 
ducting evaluations and who is specified as the recipient of the evaluation reports. 
Most of the laws (and sections within laws) include evaluation references specifying 
responsibility for conducting evaluation and dissemination of results. 

(8) Most of the laws in the health area and many of the laws in the other 
areas include wording for recommendations expected from the evaluations. Most 
recommendation language is quite specific. Recommendations are expected on a 
broad range of topics including changes in the legislation, changes in the program, 
changes in program plans, and improvement in evaluation methods and measures. 24 / 



2 3 ' According to GAO this section "directs standing committees of the House and Senate 
to review on a continuing basis, the application, administration, and execution of ■ 
those laws, or parts of laws, within its jurisdiction in order to assist the House 
and Senate in (1) its analysis, appraisal, and evaluation of the application, ad- 
ministration, and execution of the laws enacted by the Congress, and (2) its formu-- 
lation, consideration, and enactment of such modifications or changes in those laws, 
and of such additional legislation as may be necessary or appropriate." 

2 4/ Congressional Research Support and Information Services, op. cit., pp. 338-9. The 
♦0 acts requiring evaluation are summarized in the publication. 



515 



CRS - 13 

In support of assisting committee chairmen's efforts to improve the quality of 

legislative language requiring evaluation, the GAO subsequently suggested the following 

"model" of statutory evaluation requirements: 

The (head of the agency) shall submit an evaluation report, to (committees 
on appropriations and committees having legislative jurisdiction over the pro- 
gram) no later than of each year. Such report shall — 

1. contain the agency's statement of specific and detailed objectives for 
the program or programs assisted under the provisions of this Act, and 
relate these objectives to those in this Act. 

2. include statements of the agency's conclusions as to effectiveness of 
the program or programs in meeting the stated objectives, measured 
through the end of the preceding fiscal year. 

3. make recommendations with respect to any changes or additional legisla- 
tive action deemed necessary or desirable in carrying out the program 
or programs . 

4. contain a listing identifying the principal models, analyses and studies 
supporting the major conclusions and recommendations, and 

5. contain the agency's annual evaluation plan for the program or programs 
through the ensuing fiscal year for which the budget was transmitted to 
Congress by the President. 25 / 

3. Criticisms of Specifying Program Goals in Legislation . It should be noted that 
important objections have been raised to the notion of the Congress's specifying ex- 
plicit legislative program objectives and goals for program evaluation purposes. Dr. 
Harold Orlans, of the Brookings Institution, reports that complex political and plu- 
ralistic forces in the legislative process generate "...social programs [which] are 

marked by . . . multiple and contending forces Congress must serve more easily by 

, amalgam and compromise measures, which offer something to many different groups and 
1 are lent a semblance of unity by goals broad enough to embrace the spectrum of interests 
1 involved." 26/ He substantiates this notion, in part, with comments from Charles 
Schultze, based on his experiences as the Budget Bureau Director responsible for im- 
plementing the Programming-Planning-Budgeting System: 



25/ Marvin, Keith and James L. Hedrick. GAO Helps Congress Evaluate Programs. Public 
Administration Review, no. 4, July/August 1974: 331. 

26/ Dr. Orlans bases his comments on a study he prepared on "The Use of Social Re- 
search in Federal Domestic Programs," prepared for the Research and Technical 
Programs Subcommittee of the House Committee on Government Operations. They are 
in: Contracting for Knowledge: Values and Limitation of Social Science Research, 
Jossey-Bass, Inc., 1973. pp. 123-124. 



516 



"[Schultze] points out how diametrically opposed are the evaluator's need for 

a clear statement of program goals and the contrary need for vagueness in getting a 

program adopted in the first place: 

The first rule of the successful political process is, 'Don't force a 
specification of goals or ?nds.' . . . [The] necessary agreement on particular 
policies can often be secured among individuals or groups who hold quite diver- 
gent ends. . . .The Elementary and Secondary Education Act of 1965 . . . was 
enacted precisely because it was constructed to attract the support of three 
groups, each with quite different ends in view. Some saw it as the beginning 
of a large program of Federal aid to public education. The parochial school 
interests saw it as the first step in providing . . . financial assistance for 
parochial school children. The third group saw it as an antipoverty measure. . 
If there had been any attempt to secure advance agreement on a set of long- 
run objectives, important elements of support for the bill would have been 
lost, and its defeat assured. 27/ 

As a variant of this theme, the Research and Policy Committee of the Committee 
for Economic Development in a report on Improving Federal Program Performance , notes 
that program objectives and legislative statements of intent are so complex that ade- 
quate output or achievement measures should be developed only after the program has 
begun and enough time has elapsed to indicate its impacts. 28 / 

B. Better Communication for Effective Evaluation . 

Another issue, frequently treated in the literature, is the need for improved 
communication between social program evaluators and agency staff who develop program 
objectives and requirements for evaluation, in order to enhance the correspondence 
between the capability of evaluators and the expectations for evaluation. Generally 
reports on the need for better collaboration focus on the following themes: 



2 7 / Orlans, op. cit., pp. 124-5, citing: C.L. Schultze The Politics and Economics 
of Public Spending. Washington, D.C., Brookings Institution, 1968. pp. 47-49. 

28 / Improving Federal Program Performance. A statement on national policy by the 

research and policy committee of the Committee for Economic Development, Septem- 
ber 1971: p. 61. 



517 



-there is a need for social scientists and program administrators to determine 
jointly objectives of program evaluation and methodologies for research before 
an RFP is published ; 29/ 

-program evaluators who are social scientists frequently prepare long, jargon- 
ladden reports which are not written in a manner which meets policymaking needs 
and which are often overdue, causing Federal agencies to award funds for program 
evaluation to profit-making research units which prepare timely, but often inade- 
quate and sometimes invalid program evaluations; 30 / 

-Many social scientists shy away from doing program evaluation research because 
they do not consider it fruitful for generating basic information and knowledge 
required to advance the social sciences as disciplines, and the university peer 
reward structure does not value evaluation research the way it does basic re- 
search and applied research in the social sciences; 

-Federal agencies are not properly organized to administer social science research 
programs, and administrators do not understand the complexities and limitations 
of social science analytical capabilities; and 



This issue is explored in greater detail in Part IV of this report. However 
it should be noted that literature on this theme is both vituperous and construc- 
tive. Reflecting the first position: Behavior Today , a newsletter distributed 
to the social science community, reports the following on discussion at a ses- 
sion of the September 1971 convention of the American Sociological Association: 

NSF's Joel Snow, OE's John Evans and NIMH's Kenneth Lutterman painted a 
bright future for social science funding, but emphasized priority on ap- 
plied and evaluation research. Evaluation research, suggested Peter 
Rossi from Johns Hopkins, in a breezy but blunt rebuttal, often amounts 
to research on nonsense. 'Sociologists ought to have the guts to say so.' 
Politicians, Rossi claimed, controlled the conception of the poverty pro- 
gram, the community health centers, which he called silly, and NSF's new 
Rann program which he dubbed 'a WPA for engineers and political scientists'. 
Sociologists, he explained to the government representatives, are hesitant 
to spend three years of their lives evaluating such programs. There's 
something wrong with social science and the federal government,' said 
Rossi. 'We never get a chance to define the problem. The problems, even 
the social scientists themselves, are defined by the reigning politicians.' 

See also: Program Evaluation: Review of Major Issues and Literature, a 
report prepared with the Instructions of the Hon. Roman L. Hruska. By Genevieve 
J. Knezo, Science Policy Research Division, Congressional Research Service, Li- 
brary of Congress, January 31, 1972: 247 pages. (Reproduced and distributed by 
Law Enforcement Assistance Administration, United States Department of Justice, 
May 19, 1972.) 

For additional information on this issue, see Part IV of this report. 



518 



-program evaluators must become more sensitive to the political considerations 
which impact on the initiation and use of program evaluation; they should con- 
sider evaluating the political impacts of a program or at least be sensitive t 
the political values impacting on a decisionmaker who may or may not use the 
evaluation report. 31 / 



31/ The theme of political/bureaucratic barriers and resistance to the use of 
evaluation and social experiments surfaced repeatedly during the Dartmouth/OECD 
Conference on Social Research and Public Policies, September 1974. In summary 
it was noted that the same political consensus or conditions which permit the 
conduct of experiments or evaluations produce independent political consensus to 
adopt a program even before the results of the research are available. Similarly, 
an evaluation or experiment which generates positive results for the intervention 
strategy will not lead to large-scale adoption of the program if political condi- 
tions do not permit it. Good evaluations which yield negative results for pro- 
gram effectiveness may not be used because legislators and agency program managers 
don't want to see their pet project cut. Good or bad evaluations which yield 
positive results for program effectiveness may spell the demise of a program since 
the program may be perceived as having achieved its purposes . 

For additional information on these topics see: Campbell, Donald T. Assessing 
the Impact of Planned Social Change, Background Paper for the Dartmouth OECD Seminar 
on Social Research and Public Policies, September 13-15, 1974, passim; Weiss, Carol 
H. Where Politics and Evaluation Research Meet. Evaluation, v. 1, n. 3, 1973: 
37-45; and Boeckmann, Margaret. Policy Impacts of the New Jersey Income Mainte- 
nance Experiment. A paper presented at the American Sociological Association 
Meetings, Montreal, August 25-29, 1974, mimeo: 28 pp. 

An in-depth study on how decisionmakers use and do not use scientific and 
technical information, including social science information, is being conducted 
by Nathan Caplan at the Institute for Social Research University of Michigan. 
Kor a short description, see: Science Is Seldom Put to Good Use by U.S. Officials. 
ISR Newsletter, v. 2., n. 1, Spring 1974: 2, 8. 

Recommendations for social scientists to devote more attention to the broker 
concept in communications — to obtain a better awareness of policymakers' needs 
and modes of behavior, decisionmaking requirements, and constraints may be found 
in: Lane, Robert. Social Science Research and Public Policy. Policy Studies 
Journal, v. 1, n. 2, Winter 1972: 111, Jones, Ernest. Law, Political Science, 
and Policy Studies. Policy Studies Journal, Autumn 1973: 56-71; Elkin Stephen L. 
Political Science and the Analysis of Public Policy. Public Policy, v. 22, n. 3, 
Summer 1974: 399-422; Shonfield, Andrew. The Social Sciences in the Great De- 
bate on Science Policy. Minerva, July 1972: 426-438; Horowitz, Irving Louis. 
Social Science Mandarins: Policymaking as a Political Formula. Policy Sciences, 
v. 1, Fall 1970: 339-360; and Levin, Martin A. and Horst D. Dornbusch. Pure and 
Policy Social Science: Evaluation of Policies in Criminal Justice and Education. 
Public Policy, v. 21, n. 3, Summer 1973: 383. 



! : 



519 



CRS - 17 

C. The Integration of Program Evaluation Into Program Management . 

Recent congressional reviews of program evaluation in specific agencies and social 
science assessments of ways to improve the use of program evaluation research stress 
that evaluation would be more useful if these activities were better integrated into 
policy planning and program management. It is asserted that social scientists, pro- 
gram evaluation researchers, and program evaluation managers must take steps to enhance 
this cross-fertilization of the conduct and utility of such research. 

1. Incorporation of Evaluation Into Agency Program Planning and Budgeting . The 

absence of a link between evaluation research and decisionmaking, especially budgeting, 

is probably the most important impediment to effective use of evaluation research. 

Buchanan, Horst and Scanlon recently completed an examination of the evaluation and 

decisionmaking systems of the Department of Labor and the National Institute of Mental 

Health. On the need to incorporate evaluation into decisionmaking in these agencies, 

and by implication into others, they observe: 

[An organizational adjustment is needed to establish] a formal relationship 
between evaluation planning and an agency's decision processes. In the ab- 
sence of such a relationship, the same types of evaluation questions are 
raised year after year in different decision contexts — issue papers, strat- 
egy papers, and legislative proposals — without any apparent contribution or 
resolution by the evaluation system. If the evaluation information is ever to 
penetrate agency operations, the evaluation models will have to become an inte- 
gral part of the decision process. This will require training of key staff — 
evaluation planners, program managers, policy-makers — in the design and use 
of the evaluation models. 32 / 

Buchanan and Wholey observe that Federal agencies have a responsibility for in- 
corporating evaluation into program management systems and that "the establishment of 
an evaluation system usually requires the integration of several systems relating to 
planning, management, and data collection." 33_/ Such integration necessitates 



32 / Buchanan, Garth Pamela Horst, and John Scanlon. Improving Federal Evaluation 
Planning. Evaluation, v. 1, no. 2, 1973: 90. 

33 / Buchanan, Garth N. and Joseph S. Wholey. Federal Level Evaluation. Evaluation, 
Fall 1972: 22 and passim. 



71-867 O - 76 - 34 



520 



CRS - 18 

"coordination among several organizational units and sometimes ... changes in the ad- 
ministrative structures themselves," actions which are not easily taken by entrenched 
Washington bureaucrats. Walter Williams, a former chief of the Research and Plans Di- 
vision, Office of Economic Opportunity, evaluated the organizational problems surrounding 
the failure of major social program evaluations of HEW, HUD, the Department of Labor 
and the Office of Economic Opportunity in the late 1960s. _34/ He made the following 
administrative recommendations to fashion closer functional authority between research 
and implementation: (paraphrase) 

(1) the central research analyst should be given more power to generate research 
requirements and policy analysis required to develop recommendations regarding pro- 
gram funding and implementation; 

(2) the analyst should be given more money to conduct long-range social experi- 
mentation research and evaluation; 

(3) more feedback loops should be built between the field and the central office 
to overcome some problems of implementation; 

(4) the central analyst should have capability to fund social science research 
which leads to recommendations on program improvement rather than merely program 
achievement; and 

(5) more attention should be paid to stressing outcome objectives in implementa- 
tion rather than input measures. 35/ 



_3jty Williams' major points were summarized in an article reviewing major recent criti- 
ques of program evaluation and social experimentation methodology: "(1) the major 
failure of social action programs in the 1960s was in not being able to move from 
the design of the program to the implementation of the program, particularly in 
the field; (2) the major deficiency in data was that it was macronagative (showing 
how many blacks were below the white age group in certain reading achievement tests) 
instead of micropositive (how do you improve that reading level) ; (3) the central 
analysis focused too much on what was happening in Washington and not enough on 
what was happening at the local level; and (4) the social science research community 
is not structured to respond to the analytical needs of social agencies - its re- 
ward structure mitigates against research directly in support of policy analysis." 
(Robert E.C. Wegner. Evaluating the Evaluators. Public Administration Review 
[January /February 1973]: 85-6). 

35 / Williams, Walter. Social Policy Research and Analysis: The Experience in the 
Federal Social Agencies. New York, American Elsevier Publishing Company, Inc. 
[1971] pp. 180-1. 

Weiss 's report on administrative obstacles to effective program evaluation in 
the National Institute of Mental Health concludes: 

"...While not all of the impediments can be planned away in advance, certain 
basic [organizational] conditions should be present before evaluation grants are 



521 



CRS - 19 

2. Tne Development of Evaluation Work Plana . A second class of problems dis- 
cussed in relation to incorporating program evaluation into management is the need for 
an agency to develop comprehensive work plans for evaluation. This includes setting 
priorities among programs to be evaluated and designing a succession of evaluation 
studies which provide answers required to determine the adequacy of the program over 
time. Wholey, et. al. summarized this problem in the 1970 study: 

With few exceptions, Federal agencies have not had adequate work plans for 
evaluating their major social programs. 

The lack of evaluation work plans often resulted in haphazard letting of 
unrelated contracts to evaluate the same program. Contractors designed their 
own studies, formulated their own questions, and naturally produced data and 
findings that were not comparable with each other. 36 / 

3. LEAA: An Illustration of These Deficiencies . A congressional investigation 
of the administration of the block grant program of the Law Enforcement Assistance 
Administration attributes major failures in program planning and evaluation to the ab- 
sence of a well-formulated plan for evaluation. Many of the findings of this study 
probably also characterize the social program evaluation mechanisms of other Federal 
agencies. Because of their generality and potential contribution to additional legis- 
lative inquiry on evaluation aechanisas , these findings will be sunnarized: 

-Although the LEAA has authority under its enabling legislation to "conduct 
evaluation studies of the programs and activities assisted under the title," 
the agency has done little toward making its own evaluation of the effective- 
ness of programs or projects funded with block grant funds. 37 / 

-One of the major deficiencies LEAA encounters in developing adequate evaluation 
plans is that because of poor information systems, the agency lacks knowledge 



made. Among the most important are: (1) clarity of purpose for the study, (2) 

a well-defined and relatively stable program, (3) administrative support, and 

(4) an able research staff who give a substantial proportion of time to the study." 

(Weiss, Carol H. Between the Cup and the Lip Evaluation, v. I, no. 2, 1973: 

55.) 

36 / Wholey, et. al. , Federal Evaluation Policy, op. cit. p. 35. 

37 / U.S. Congress. House Committee on Government Operations. The Block Grant Program 
of the Law Enforcement Assistance Administration. Part 1, Hearings before a Sub- 
committee of . . . July 1971. Washington, U.S. Govt. Print. Off., 1971. p. 143. 



522 



about the disbursement of grant funds and the purposes to which these funds will 
be spent. This deficiency should be rectified with an improved grant management 
information system. 38/ 

-Equally, if not more important in terms of evaluating management performance 
and capabilities, neither LEAA nor State agencies which administer the programs 
have formulated standards for evaluating program progress, success, or failure. 
Thus, the programs are unevc-.^ated, unaudited, and incapable of being measured 
as to performance and progress due to the lack of goals or standards. 39 / 

-Since LEAA does not maintain an inventory of grant programs and lacks a central- 
ized evaluation plan and standard, its efforts sometimes duplicate those of other 
agencies. 40/ 

-Many LEAA funds are administered directly by State Planning Agencies, (SPA). 
The lack of program goals and evaluation standards and guidelines prevents SPAs 
from adequately administering and evaluating programs. LEAA has the responsi- 
bility for providing SPAs with manuals and guidelines for evaluation but has 
not adequately responded to this need. 41 / 

4. The Contribution of Evaluation Work Plans to Evaluations Performed at the 
State Level . Federal assistance to State and local jurisdictions for developing evalu- 
ation plans, objectives, methodologies, and information-dissemination systems seems all 



38/ Specifically: ". . . .LEAA [lacks] knowledge of the purposes, goals, and specifi- 
cations of the thousands of State and local projects financed by its block grants. 
This informational void, still unfilled as LEAA completes its fourth year of opera- 
tion, makes LEAA powerless to modify programs to avoid deviations and to follow 
congressional intent. A grant management information system and a reference ser- 
vice, both of which should alleviate these and other monitoring problems of LEAA, 
are currently being devised by outside consultants under contract to the agency." 
(U.S. Congress. House. Committee on Government Operations. Block Grant Program 
of The Law Enforcement Assistance Administration. Twelfth Report by the Comm. on 
Government Operations. House Report. No. 92-1072. May 18, 1972. Washington, D.S. 
Govt. Print. Off., 1972. p. 8.) 

39/ Idem. 

40/ Ibid., p. 61. 

4V Specifically: Statement of Mr. Ahart. "In the SPA Guide for 1969 (a manual is- 
sued by LEAA to the State Planning agencies for guidance on application awards, 
and administration of planning and block grants). LEAA stated that it would is- . 
sue guidelines suggesting appropriate procedures, techniques, and measures for 
evaluating the contribution to crime control of the block grant projects and ex- 
penditures. The guide provided that the State pi arming agencies, pending issuance, 
of the guidelines, outline in their 1969 State plans a tentative program for pro- 
ject evaluation and the measurement of overall plan performance. ... We were ad- 
vised by the LEAA that it had not issued guidelines to the State planning agencies 
on evaluation methods because of a shortage of manpower. Also we noted that on 
occasions, information or guidance has been requested from LEAA on monitoring and 
evaluation methods, and LEAA has been unable to provide the assistance." (The 
Block Grant Programs of the Law Enforcement Assistance Administration, (Part 1), 
hearings, op. cit., pp. 143-4 



523 



the more imperative since State and local governmental units have been given greater 
responsibility for program administration and accountability under revenue sharing. A 
few agencies have encouraged developments in this area, notably Housing and Urban De- 
velopment and the Office of Management and Budget. 42/ In a partial effort to assist 
non-Federal authorities in performing required audit functions, the General Accounting 
Office prepared a report in June 1972, titled Standards for Audit of Governmental Or- 
ganizations, Programs, Activities, and Functions , 54 pages. This report does not how- 
ever, provide standards for program evaluation. The GAO is now preparing a program 
evaluation guidebook, expected to be released in the Spring 1975. 

5. Proposals for Comprehensive Evaluation Plans . A number of agencies conducting 
program evaluation research have begun to develop coherent evaluation work plans to 
guide both internal agency evaluators and extramural evaluators. 43/ As will be noted 
below, the Office of Management and Budget is now encouraging activities in this di- 
rection. It may be of interest to the Congress to inquire into whether other Federal 
agencies are taking steps to provide for systematic development and use of evaluation 
work plans. Who ley, et. al. made a number of recommendations treating this issue. Some 
of their observations are reproduced below: 



42/ See: Ninth Conference on Management Analysis in State and Local Government, No- 
vember 2 and 3, 1972, sponsored by Institute of Public Service, University pf 
Connecticut. Among the papers were: Program Evaluation: Pitfalls and Potential- 
ities, Dr. Jack Carlson, OMB; Case History Application of Program Evaluation, Keith 
Marvin, General Accounting Office. The most useful study in this respect is the 
HUD funded report: Hatry, Harry P. et. al. Practical Program Evaluation for State 
and Local Government Officials. Washington, D.C., The Urban Institute, September 
1972. 134 pp. 

43/ For an introduction to this topic, see R.A. Levine and A. P. Williams, Jr., 

Making Evaluation Effective: A Guide. A report prepared for the Department of 
Health, Education, and Welfare. Santa Monica, The RAND Corporation, May 1971, 40 
pp. (R-788-HEW/CMU.) Fred Spaner and Charles Winder, National Institutes of 
Mental Health. The Perspective of National Program Development and Administration: 
Program Needs and Evaluation Activities. Unpublished, February 5, 1971. Presented 
at American Psychological Association Convention September 1971. Quotation from 
Federal Evaluation Policy , op. cit., pp. 114-115. 



524 



CRS - 22 
EVALUATION PLAN 

At the beginning of each fiscal year, each agency should develop or update 
a two to three-year evaluation work plan stating which studies are in progress 
and which are projected. Approval of this plan should be required at the be- 
ginning of each fiscal year as a condition for authority to spend evaluation 
funds. This plan should be prepared by the agency-level evaluation staff in 
cooperation with policy makers, budget staff, program managers and operating- 
level evaluation staffs. Evaluation planning is a continual process: when 
the agency appropriation is known, the evaluation plan should be reassessed 
in light of firm budget figures. 

Since useful evaluation studies are likely to be costly, agencies should 
place emphasis on feasible studies of major programs where the value of the 
findings would outweigh the costs. 44 / 

6. Proposals for a "Preassessment of the Evaluability of Programs ." As a variant 
of this theme, researchers at the Urban Institute suggest that the conduct and use of 
program evaluation will be enhanced if federal agency administrators and program evalu- 
ation researchers conduct a "preassessment of the evaluability of a public program." 
The objective of the assessment would be to select appropriate research strategies for 
evaluation. The following questions, addressing the assumptions of the program, its 
data base, and the probable use of the evaluation in decisionmaking processes, suggest 
a sequence of evaluation steps and criteria to be used in determining the "evaluability 
of a public program" and priorities among different types of evaluation strategies: 

Are the problems, intended program interventions, anticipated outcomes, and 
the expected impact sufficiently well defined as to be measurable? 

In the assumptions linking expenditure to implementation or intervention, 
intervention to the outcome anticipated, and immediate outcome to the ex- 
pected impact on the problem, is the logic laid out clearly enough to be 
tested? 

Is there anyone clearly in charge of the program? Who? What are the con- 
straints on his ability to act? What range of actions might he reasonably 
take or consider as a result of various possible evaluation findings about 
the measures and assumptions discussed above? 45/ 



44/ Federal Evaluation Policy, op. cit., pp. 114-5. 

45 / Horst, Pamela, Joe N. Nay, John W. Scanlon, and Joseph S. Wholey. Program Manage- 
ment and the Federal Evaluator. Public Administration Review, v. 4, July/August 
1974: 307. (Part of PAR/special "Symposium on Program Evaluation.") (Footnote 
continued on p. 23.) 



525 



D. The Need for Better Dissemination of Information on Program Evaluations Supported 
Under Federal Agency Auspices 

Several issues are subsumed under the general heading of the need for improved 
program evaluation information dissemination systems. Among them are: the need for 
researchers to share experiences in developing methodologies to hasten refinement of 
program evaluation techniques; to eliminate duplicate funding of methodologies and 
e\aluations of programs; and to insure distribution of program evaluation reports to 
individuals within an agency and to individuals outside of the agency whose program 
responsibilities intersect with the program being evaluated. 

Howard R. Davis, chief of the Mental Health Services Research and Development 

Branch of the National Institute of Mental Health, has summed up these concerns: 

Increasing numbers of investigators are working vigorously on methods of measure- 
ment and analysis in evaluation; but one researcher hardly knows what the others 
are doing. Duplication unavoidably is occurring. Incompatible counterparts of 
evaluation systems are needlessly re-invented. . . . Better methods of ex- 
changing information [should] be developed as soon as possible. Not only would 
each person producing evaluation knowledge know what others are doing, not only 
would those needing to implement evaluation programs have a valuable perspective 
of what is going on elsewhere, but the concepts of evaluation. . . would receive 
continued clarification. . . . ^ 6 / 

Perhaps as a partial response to this need, the National Institute of Mental Health 
awarded a grant to the Program Evaluation Project, Minneapolis Medical Research Founda- 
tion, to establish, on an experimental basis, Evaluation , the journal in which this 
statement is found. 



The National Advisory Council on Educational Professions Development made a 
related recommendation for the administration of each specific program evaluation: 
"Evaluation research plans can fail in implementation just as programs can Jail. 
Time must be taken to 'walk through' evaluation research designs by applying them 
to simulated program data, operational settings and decision questions, before 
applying the evaluation research design to the actual program. Such a process 
could create much more understanding of various models of evaluation research and 
alert decision-makers to potential problems and results in advance." (Search for 
success , op. cit., 24.) 

*jf David, Howard R. Insights: A Solution for Crisis? Evaluation, v. I, no. 1, Fall 
1972: 3-4. 



526 



1. Investigation by the General Accounting Office . A number of recent reports 
indicate that far more systematic methods of exchanging information on program evalua- 
tion may be required. For example, the General Accounting Office published a report 
entitled "Program Evaluation: Legislative Language and a User's Guide to Selected 
Sources," in June 1973. This report was prepared to assist GAO researchers obtain pub- 
lished information and program evaluations to which they could refer in performing their 
auditing and evaluation functions for the Congress. The author describes and assesses 
30 on-going research inventory and published information sources established by Federal 
agencies. These, among others, include the National Technical Information Service 
(NTIS) ; the Science Information Exchange (SIE); ERIC, a research report inventory for 
federally funded education research; and MEDLARS and NCHSRD (Inventories of research in 
progress or published related to medical and mental health services, respectively). 

The report indicates the following inadequacies: 

Despite [the] rapid growth in evaluation studies, information as to who is 
conducting which studies with what findings is not readily available. No single- 
source document or retrieval system identifies planned, ongoing, and completed 
evaluations. No single source describes the findings and conclusions of completed 
evaluations. In fact, no single functional area (such as health, education, or 
training) has such complete coverage of evaluation studies. Since program evalua- 
tion provides perhaps the most comprehensive, objective, and quantitative sources 
of Information on governmental program results, it is Important to the government 
and to the public that these results be identified and made available. Most non- 
defense evaluations are public information and need to be listed and described. 

A number of computer retrieval systems and reference documents listing a 
large number of studies, including evaluation studies, are available. However, 
knowledge is generally lacking as to which information sources exist, which sources 
contain evaluation studies, how comprehensively evaluation studies are described, 
and how to obtain information from such sources. This chaotic situation makes it 
a long-term task to develop, maintain, and distribute a comprehensive list and 
description of planned, ongoing, and completed evaluation studies in all functional 
areas . Ujl 

2. NIMH/UCLA Project on Data Bank of Program Evaluation (DOPE) . It should be 
noted that the Health Services and Mental Health Development Branch of the National 



47/ Program Evaluation: Legislative Language and a User's Guide to Selected Sources, 
op. clt., p. s l. 



527 



Institute of Mental Health is funding a joint project with UCLA to establish an easily 
accessible computerized information system on program evaluation, called the Databank 
of program Evaluations (DOPE). The system, expected to be in operation in early 1975, 
will enable all researchers who have access to the ARPA computer network (of the Ad- 
vanced Research Projects Agency) to obtain an abstract of the following information for 
each program evaluation report and secondary source document included in the system: 

-citation^ 

-status of the program success, 

-condition treated 

-age, Be*, race and income characteristics of the sample population, 

-sample size, 

-site, 

-treatment method, 

-characteristics of the study design, 

-what is measured, 

-what measures are used, 

-how data Is collected, and 

-conclusions. 

Evaluation reports will encompass the following topics! alcohol abuse, divorce, juvenile 
delinquency, neuroses, sexual deviancy, suicide, welfare, health problems with mental 
health outcomes, criminality, drug ahu.se, mental retardation, psychoses, socioculfcural 
problems, unemployment, and mental problems of the aged. Three types of search are 
being used to Identify reports of evaluations: (1) journal articlesj (2) published and 
unpublished reports recommended by experts in each subject field, and (3) materials 
from other computerized data banks, such as Medlars, Psychological Abstracts, and Socio- 
logical Abstracts. 48/ 



^8/ Wilner, Daniel M., Robert W. Hetherington, Ellen B. Goid, Daniel H. Ershoff, and 
Cedric F. Garagliano. Databiink of frogram Evaluations. Evaluation, V. I, no. 3, 
1973: 3*6. (The project is supported under grant IR 12-MH-2276.) 



528 



PART III. IMPROVED COORDINATION OF 

PROGRAM EVALCATION IN THE EXECUTIVE 

AND LEGISLATIVE BRANCHES 

A; Issues Su rrounding the Need to Incorporate Program Evaluation Findings Into 

Executive Branch Budget Accesses 

Two topics are included under this heading of improved coordination of program 
evaluation in the executive branch. The first relates to ways to further incorporate 
program evaluation findings into executive branch budget processes. The second deals 
with imprcved coordination of the administration and use of program evaluation in the 
executive branch. 

The c?r.ly government -wide requirements fcr agencies to incorporate program 
evaluation data into an jual agency budget proposals are found in Circular A-ll, issued 
by the Office of Management and Budget. The term "program evaluation" is used very 
loosely i R this circular, since costs anc. benefits or program accomplishments or ont- 
?utF . are to be -stated directly in dollar terms or in number of persons affected, 
rather than in more hard to measure terms of real, but mostly intangible, program 
accomplishments. For instance, data to be submitted for Federal Income Security 
Programs consist of three types: benefit outlays, number of beneficiaries and 
average monthly benefits, number of individuals benefited by the outlays and number 
of primary beneficiaries. These data are required of separate target groups cate- 
gorized as aged or retired; disabled; mothers; surrvivinf, spouse; other survivors; 
unemployed and others. (Sec. 49.) These data requirements differ considerably from 
requirements for most agency program evaluations, which are designed to determine 
whether a program should be modified, terminated, or continued. Evaluation require- 
ments under Circular A-ll also do not yield information to determine which among 
alternative social programs best serves a particular target group or meets an accepted 
national need. These requirements have also been videly criticized as inappropriate 



529 



to the budgeting process in both executive and legislative branches since almost no 
attempt is made by the Office of Management and Budget to integrate the results of 
program evaluations and program evaluation information to the annual budget process. 49/ 
Criticism has been heard also that what information the executive branch does collect 
is withheld from the Congress all too frequently on the grounds of executive privilege. 
Commenting on this point in 1970, (when Office of Management and Budget was called 
the Bureau of the Budget), the Joint Economic Committee said: 

Program Memoranda, containing quantitative evaluation of the benefits and 
the costs of individual programs as well as other evaluative information, have 
been retained ... on grounds of executive privilege. Special Analytic Studies, 
presenting the results of detailed study of the efficiency and equity consequences 
of alternative means of obtaining objectives, are from time to time released, but 
only after the executive has made a policy decision and then only in a form 
designed not to upset the decision. Program and Financial Plans, with program 
budget projections, are held within the executive on grounds of lack of data 
and the unreliability of estimates. While the list of issues on which the 
Bureau of the Budget is requesting analysis in the planning and budgeting cycle 
has been released to the Congress through the subcommittee, the results of the 
analyses of these issues have been withheld . . . 

... The lack of congressional access to much of this information and analysis 
seriously hinders legislative efforts to scrutinize and control the budget. The 
widespread use of executive privilege to retain the results of program evaluation 
and analysis is inimical to the development of an open and responsive political 
system in which public policy is made by elected political leaders rather than 
by an anonymous and powerful bureaucracy .5°/ 

And although 0MB asks agencies to describe generally how programs meet national 

objectives, termination of requirements for agencies to adhere to the Planning, 

Programming, Budgeting system, in effect, removed requirements for agencies to analyze 



^ I Allen Schick, "An Analysis of Proposals to Improve Congressional Control of 
Spending." In U.S. Congress. Joint Economic Committee. Subcommittee on 
Budgeting, Management and Expenditures. Improving Congressional Control Over 
the Budget: ACompendium of Materials. March 27, 1973. Washington, U.S. 
Govt. Print. Off., 1973. pp. 223-4. (Committee Print.) 

_/ U.S. Congress. Joint Economic Committee. Subcommittee on Economy in Government. 
Economic Analysis and the Efficiency of Government. Report. February 9, 1970. 
Washington, D.C., U.S. Govt. Print. Off., 1970. p. 9. (Joint Committee Print.) 



530 



program accomplishments and deficiencies In relation to program planning. 51/ 



51 / General Information on removal of requirements for PPB, stemming from admini- 
strative problems and difficulties in obtaining required information, are 
reviewed in: Schick, Allen. A Death In the Bureaucracy: The Demise of Federal 
PPB Public Administration Review, vol. 33, March/April, 1973: 146-156. 
Specific details on requirements for submitting evaluation data in budget 
proposals and on the evolution of PPB requirements were supplied by Mr. Joel 
Anderson, Office of Management and Budget: 

According to Mr. Anderson the most important 0MB directive on program evaluation 
is Bulletin No. 68-9, issued April 12, 1968, supplemented specifically by 
Bulletin No. 68-9, supplement No. 1, July 17, 1969, and supplemented generally 
by Circular No. A- 11, usually revised and re-issued each year. Bulletin No. 
68-9 and supplement No. 1 treat the Planning-Programming-Budgeting System. 
Circulars A-ll treat preparation and submission of annual budget estimates. 

These 0MB directives treat program evaluation only very generally within the 
context of "programming" as part budgeting/appropriations submissions to the 
Congress. Effectiveness is treated in this sense from an economic approach. 
The "programs" which are to be evaluated for the purposes of preparing data 
required by these directives are general budget categories, not specific oper- 
ational agency programs. 

President Richard M. Nixon issued a Memorandum to the heads of Federal departments 
and agencies, May 25, 1970, In which he outlined the importance of evaluating 
effectiveness and using program evaluation to underpin agency planning and 
budgeting activities. He outlined general criteria which the required program 
evaluations were to meet. 0MB circular A-ll, 1971 (With transmittal Memorandum 
No. 38) , reflects the need for agencies to meet these general criteria in 
preparing budget estimates. However, the criteria outlined in the Circular are 
discretionary and diverge somewhat from the specific requirements under the 
rescinded PPB guidelines. ("Analysis of Changes in Office of Management and 
Budget Circular No. A-ll, Revised June 1971, Comparison of Old Sec. No. 
24.1-24.6 to 24.1-24.5, p. 3.) 

Program evaluation requirements in the current circular, A-ll, Revised, 
Transmittal Memorandum No. 39, June 12, 1972, are similar to the generalized 
requirements of the 1971 circular. These requirements are outlined in 
paragraphs 24.2-24.3 (Analysis and Justification of Programs and 36.1-2 

Narrative Statements on Program Performance.) The material requested to 
fulfill requirements for "Analysis and Justification of Programs" supports 
the first part of the budget request; material required to meet "Narrative 
Statements on Program and Performance" in the circular Is used for detailed 
budget appendix. Mention is made of a number of techniques for conducting 
evaluations, and for relating program objectives and inputs to program benefits 
and social benefits. However, the Circular seems to leave It up to the discretion 
of the agency director to use techniques which are most appropriate in evaluating 
the effectiveness of Its programs. 

Specific "evaluation" requirements for programs falling under separate functional 
areas, such as manpower, health, and education, are also detailed in Circular A-ll. 



531 



B. Proposals to Give the Office of Management and Budget Program Evaluation 
Responsibilities 

A number of proposals have been made to give the Office of Management and 

Budget explicit coordinating and governing responsibilities for program evaluations 

conducted by agencies of the Government. Joseph S. Wholey, et. al., examined this 

issue in a 1970 report which analyzed the evaluation activities of four Federal 

agencies. They reported: "The executive branch has no system for Identifying 

major questions to be answered through evaluation, for ensuring that high priority 

studies were carried out, or for ensuring that significant evaluation findings 

reached appropriate decisionmakers." They recommended that: the Bureau of the 

Budget (now called 0MB) (a) require agencies to develop two-to-three year evaluation 

plans and to use evaluation information in budgetmaking and legislative review; 

(b) evaluate the quality of evaluation techniques used by different agencies; 

(c) conduct cross-agency evaluations "to compare the effectiveness of related 
Federal programs in achieving common objectives;" and (d) enlarge budget allocations 
for evaluation staff and resources. 52/ 

C. The Proposal to Establish a Federal Program Evaluation Office in Either the 
Office of Management and Budget or in the Executive Office of the President 

Laurence E. Lynn, Jr., who served as Assistant Secretary for Planning and 

Evaluation in the Department of Health, Education and Welfare, between 1971 and 

early 1973, made the most recent proposal in this respect. In order to overcome 



^2/ Federal Evaluation Policy, op. cit., 24, 58-61. Similar recommendations have 
been made by the Committee for Economic Development in its report Improving 
Federal Program Evaluation, op. cit., 59 and by the President's Advisory 
Council on Executive Organization. Memorandum for the President of the United 
States on Establishment of a Department of Natural Resources [and] Organi- 
zation for Social and Economic Programs (including background memoranda and 
attachments). February 5, 1971, Appendix 5. 



532 



CRS - 30 

Che many inadequacies of current program evaluation practices, he proposed the creation 
of a Federal Program Evaluation Office in either the office of Management and Budget or 
in the Executive Office of the President to "[supervise] the evaluation activities of 
federal program agencies, [to conduct] a program of research on evaluation methods, and 
[to produce] an annual evaluation report. "53/ Specifically the proposed Federal Pro- 
gram Evaluation Office (FPEO) would: 

. . . determine how to apportion the funds among the FPEO and the agencies in 
accordance with a comprehensive plan .... 

... be directly responsible for, and retain full authority over, all program 
impact evaluations. Adequate qualified staff would be maintained to do the 
necessary planning, write requests for proposals, review proposals, and most 
important, closely monitor the work of the evaluation contractor or grantee. 

... be responsible for overseeing the development of plans for program strategy 
evaluations and project evaluations .... 

. . . prepare the annual evaluation report summarizing evaluation findings and 
work in progress .... 

. . . operate a system for rating the performance of evaluation contractors and 
grantees and for making the ratings available to evaluation officials. 

— [develop a priority framework] to guide the allocation of evaluation 
resources ... in recognition of resource limitations, the potential value of 
possible valuations, and the availability of evaluation capacity. 54/ 

D. Current OMB Initiatives in Program Evaluation 

As noted above, the OMB has stepped up its evaluation activities in order to 
improve the management and use of evaluation research. In a recent article two OMB 
officials, who were then responsible for directing the agency's program evaluation 
activities, outlined the specific objectives of the present OMB effort. These 
objectives, directed at each Federal agency, are: 



53 / Lynn, Laurence E. Jr. A Federal Evaluation Office? Evaluation, v. I, no. 2, 
1973: 29, 62. 

54/ Ibid., pp. 62,96. 



533 



(a) development of built-in evaluation capability from the inception of each 
new Federal program; (b) adequate staff to assure quality of output; (c) 
sufficient planning to focus attention on the high pay-off areas; (d) follow-up 
procedures to assure that results are utilized (e) organizational alignments 
to assure full integration with the policy decision process and related and 
supportive activities; and (f) a proper balancing of long-term and short-term 
efforts (in-house and external) to meet both quality and timeliness needs. 55/ 

Briefly, OMB's evaluation activities consist of the following: (1) establishment 
of a Division of Evaluation and Program Implementation under the Associate Director 
for Management and Operations; the Office is directed by Mr. Clifford W. Graves; 
(2) establishment "on a very informal basis of a panel of senior Federal executives 
to serve as a forum for the mutual exchange of ideas and developments within the 
Federal evaluation community;" (3) initiation of a "... comprehensive survey of 
Federal evaluation programs that will inventory and catalogue ... such activities;" 
(4) development of a "cross-agency budget review to assess the adequacy of funding 
levels for evaluation programs in a number of Federal agencies;" (5) encouragement 
of all agencies to prepare "formal plans on a periodic basis to direct and guide 
individual revaluation] studies to high priority topics;" (6) review of legislative 
initiatives "to assure that adequate provision is made for evaluation" and for 
establishment of an appropriate evaluation capability for the program; (7) incor- 
poration of evaluation priorities into the management by objectives effort; and 
(8) monitoring, on a selective basis, of evaluation studies of particular interest 
to OMB and the White House. 56/ 

As a first step in carrying out these responsibilities, the Evaluation and 
Program Implementation Division of OMB conducted a survey of federal agency 



55 / Federal Program Evaluation From the OMB Perspective, op. cit., p. 317; each 
of these objectives is fully described on pp. 314-317. 

56/ Ibid., pp. 312-314. 



534 



evaluation activities which provides information on the following points: 

- persons(s) in the agency responsible for performing different types of 
evaluations ; 

- types of activities undertaken; 

- magnitude of resources available for evaluation; 

- characterization of relationships between the production of evaluation 
information and decision processes such as management by objectives, budget 
preparation, and statistical planning; and 

- persons who can provide additional information on evaluations conducted. 

A draft version of the report was released in June. It is expected that the 

materials will be updated and disseminated more widely after additional agencies 

provide responses to the OMB questionnaire. 57/ 

E. Issues Relating to the Improvement of Legislative Branch Program Evaluation 
Capabilities 

The absence of a congressional program evaluation capability has repeatedly 
been cited as a major impediment to incorporating program evaluation reports into 
the legislative process. Two frequently mentioned themes are: (1) that the executive 
branch both conducts program evaluations and assesses the results of program evalua- 
tions without presenting reports to the Congress or to an objective critic, 58/ 



57 / Survey of Federal Evaluation Activities. By Evaluation and Program Implemen- 
tation Staff, Office of Management and Budget, June 1974, draft, loose leaf, 
various pagings. 

58 / On this point Schick reports: "... Evaluation comes too late, after the 

program has garnered strong client support and is built into the budget base. 
The classic case is summer Headstart which was serving hundreds of thousands 
of preschoolers when adverse findings were released. Popular programs are 
hard to terminate, no matter what judgments flow from the evaluations." ("The 
Pilot Testing of New Programs: An Analysis of Title IV, S. 3984," In: 
Improving Congressional Control of the Budget, op. cit, p. 278) 



535 



and (2) the Congress does not have necessary staff expertise to analyze such reports 

even if they were provided. Murray L. Weidenbaum, et. al. summarized these problems: 

Evaluative research is now a virtual monopoly of the executive branch, to 
the extent that it is used at all. If Congress is ever to oversee executive 
administration effectively, then it needs professional assistance beyond that 
provided by existing committee staffs. As imperfect as evaluation techniques 
are, at the least they would be a useful supplement to existing rules -of -thumb 
and other subjective approaches. 59/ 

In this same respect the Joint Economic Committee reported: 

The relationship of the executive and legislative branches pertaining to 
the use of executive analysis and evaluation documents is a serious problem. 
More serious, however, is the failure of the Congress to provide itself with 
an analytical capability. Currently, the Congress has neither an adequate 
capability to interpret or evaluate studies done by the executive and those 
outside of Government nor the necessary capacity to undertake policy analysis 
of its own. 60/ 

Evaluation of public welfare programs would be enhanced, the Joint Economic Committee 
reported in November 1972, by creating an evaluation agency independent of the ad- 
ministering agency and reporting directly to the Congress: 

. . . Enabling legislation usually assigns the task of evaluation to the program 
administrators. Separation of powers is a well-accepted and venerable principle, 
and its application is as appropriate here as elsewhere. Administrators are 
understandably anxious to depict their programs as successful, and evaluations 
conducted by them (no matter how conscientious they may be) cannot escape being 
suspected of bias. An independent agency accountable to Congress, should be 
responsible for evaluation. 61/ 



59/ Weidenbaum, Murray L., Dan Lark ins and Philip N. Marcus. Matching Needs and 
Resources: Reforming the Federal Budget. Washington, D.C., American Enter- 
prise Institute for Public Policy Research [1973] p. 11. 

60/ U.S. Congress. Joint Economic Committee. Studies in Public Welfare, Paper No. 
3, The Effectiveness of Manpower Training Programs: A Review of Research on 
the Impact of the Poor. A staff study prepared for the use of the Subcomm. on 
Fiscal Policy. November 20, 1972. Washington, U.S. Govt. Print. Off., 1972. 
p. 13. (Joint Committee Print.) The other studies in the series are: Paper no. 
1, Public Income Transfer Programs: The Incidence of Multiple Benefits and 
the Issues Raised by Their Receipt, April 10, 1972; Paper No. 2, Handbook of 
Public Income Transfer Program: How They Tax the Poor, December 22, 1972; 
Paper No. 5, Part 1. Issues in Welfare Administration: Welfare An Admin- 
istrative Nightmare, December 31, 1972. 

£1/ The Effectiveness of Manpower Training Programs: A Review of Research on the 
Impact of the Poor, op. cit., p. 15. 



536 



Within the last year or two proposals were made to provide the Congress with 
a program evaluation capability independent of the executive branch. Most of these 
responded to alternative ways of meeting some of the recommendations of the Joint 
Study Committee on Budget Control (■"_' One of the committee's major observations 
was that legislative determination of priorities among programs and of spending levels 
for these programs would be strengthened if the Congress coordinated more closely the 
appropriations process and the authorization process (which now fragments legislative 
review of the evaluation of the effectiveness of particular programs, evaluation of 
the merits of one program against another, and program funding).^/ 

During the 93rd Congress, both Houses directed attention to improving the process 
of determining and evaluating budget priorities to serve national needs and to 
strengthening the review of the budget process and evaluation of Federal programs. 
The Congressional Budget and Impoundment Control Act, P.L. 93-344, July 23, 1974, 
includes two sections which expand legislative program evaluation capabilities. Under 
section 701, committees are given the authority to conduct evaluations themselves, 
using pilot tests and cost/benefit analyses, if warranted, or to require agencies 
to evaluate programs and report the results to committees. Section 702 expands the 
review and evaluation functions of the General Accounting Office. Under this section 
the GAO is charged with responsibilitv for assisting committees in evaluating govern- 
ment programs, including the development of statements of legislative objectives, 



62 / The committee was created in October 1972, pursuant to P.L. 92-599, to recommend 
procedures for improved congressional control over the outlay and receipt totals 
in the budget and to help establish spending priorities in the budget. The 
committee presented its interim report to the Congress on February 6, 1973, 
"Improved Congressional Control Over Budgetary Outlay and Receipt Totals." 

63 / See section II. "Tentative Recommendations. Introductory paragraph." (In 

Interim Report by the Joint Study Committee on Budget Control. Remarks of Hon. 
Mr. Ullman on the floor of the House. Congressional Record, February 7, 1973: 
H. 843.) 



537 



methods for review and evaluation of programs, and the analysis of program results. 
The Act also authorizes the establishment of an Office of Program Review and Evaluation 
in the General Accounting Office. 64/ 

Some of the other bills considered during the 93rd Congress included provisions 
which would have made pilot testing mandatory or would have required OMB to inventory 
or to use program evaluation information in preparing budget submittals. (S. 40, Mr. 
Brock; S. 1030, Mr. Humphrey: S. 758, Mr. Beall; and S. 834, Mr. Muskie. S. 758, in 
particular, would have created a Federal program evaluation digest.) 

Related to this was a House proposal, H. Con. Res. 21, introduced by Mr. Conte 

and referred to the Committee on Rules, which would establish a "Joint Select Committee 

on Government Program Analysis and Evaluation." The Committee would be authorized to 

evaluate alternative procedures and to evaluate "the efficiency and effectiveness of 

Federal programs and activities by objective, scientific, and empirical analysis...." 

Among the alternative proposals the Committee would examine are: 

-"the establishment of an ad hoc, independent, bipartisan commission to 
study and appraise Federal programs and activities. . .of the executive branch...;" 

-"an expansion of the role and function of existing agencies and authorities 
(such as the General Accounting Office) within or outside of Congress to perform 
such analyses and evaluations ; " 

-"improved staffing of standing committees. . .through the addition of review 
specialists and persons scientifically trained and experienced in the area of 
program audit and evaluations;" and 

-"the establishment of a central staff or office as an integral part of the 
Congress charged with performing program audits, evaluations, and analyses under 
the supervision of a legislative auditor and under the general supervisory 
control of a joint committee of the Congress. . ."65/ 



£*_/ Title VTI - Program Review and Evaluation, Conference Report, H.R. 7130, 

Congressional Budget and Impoundment Control Act, passed Senate June 21, 1974. 
P.L. 93-344. 

e J_l Sec. 1 (b), H. Con. Res. 21, January 3, 1973. 



538 



F. Issues Surrounding the Role of the General Accounting Office In Providing the 

Congress with Improved Program Evaluation Capability 

The General Accounting Office's role in providing the Congress with program 
evaluations independent of the executive branch has received considerable attention 
with the last two years. Several issues are Involved: (1) Are the agency's auditing, 
accounting, and program evaluation related activities sufficient to satisfy legisla- 
tive requirements for program evaluation and evaluation of the utility of different 
program intervention strategies in meeting legislative goals? (2) Does the agency 
have sufficient authority, staffing and appropriate administrative arrangements to 
conduct program evaluations? (3) How can the GAO and the Congress work more closely 
in developing program objectives and legislative language which meets requirements 
for the conduct of program evaluation? 

1. Basis of GAO's Statutory Authority for Program Evaluation. Comptroller 
General Elmer B. Staats, noted the basis of the agency's statutory authority for the 
conduct of program evaluation in a recent article: "While the GAO has always construed 
the Budget and Accounting Act and the Legislative Reorganization Act of 1946 to include 
this authority, the Legislative Reorganization Act of 1970 made it quite explicit. 
This act, in brief, directed that GAO, either on its own initiative or at the request 
of committees of the Congress, make studies of the costs and benefits of existing 
programs. "66/ He continued that the agency has actively carried out its responsi- 
bilities under these acts: 

For the past 5 years, GAO has given high priority to the evaluation of Federal 
programs to the point where approximately 30 percent of Its professional staff 
of 3,250 is now engaged in evaluations and studies with this objective. 67/ 

2. Evolution of GAO's Program Evaluation Activities. Issues have been raised 
regarding the nature of GAO's evaluation activities. For instance, Senator Roth, 



66 / Staats, Elmer B. Improving Congressional Control Over the Federal Budget. The 
GAO Review, Summer 1973: 11. 

6? / Improving Congressional Control Over the Federal Budget, op. cit.: p. 11. 



539 



CRS - 37 

consenting upon responses received in his staff survey of program evaluation practices 

noted: 

A rather small portion of the executive departments, and an even smaller 
portion of the independent agencies indicated in response to our inquiry that 
GAO was actively or regularly Involved in evaluating the substantive accom- 
plishments of their programs. They also stated that the Comptroller General's 
interest in their programs was quite often of a fiscal-procedural nature.68/ 

Continuing, he recommended that the GAO's program evaluation responsibilities and 

those of the other congressional information support agencies, including the Library 

of Congress and congressional committee staffs, should be expanded to include more 

program evaluation impact analysis: 

It is clear that GAO has plenty of work to do and does much of it effectively. 
However, Congress needs to have more independent evaluation of the impact of 
Federal governmental activity by GAO, the Library of Congress, its own com- 
mittees, or perhaps by some other body. The Legislative Reorganization Act of 
1970 clearly assigns to the Comptroller General and the Library of Congress 
additional responsibility to perform substantive evaluations. 69/ 

In answering questions posed by critics of GAO's program evaluation efforts, the 

Comptroller General admits that "GAO still has much to learn," but that "overall good 

progress is being made." "Evaluation of government program results," he continues, 

"is an art about which all of us have much to learn." The agency faces two major 

difficulties in conducting evaluations, "particularly in the social action areas:" 

Not the least of these are (1) the lack of clearly, specifically stated 
program goals and objectives and (2) the lack of reliable data on performance 
and effects or results of program operations. 70/ 

In another article Assistant Comptroller General Ellsworth H. R. Morse reported 

that the agency is attempting to improve training and administrative procedures to 

assist GAO in meeting its program performance and evaluation responsibilities: 



68 / Public Program Analysis and Evaluation for the Purposes of the Executive and the 
Congress, op. cit.: S9026. 

69/ Idem. 

70/ Improving Congressional Control Over the Federal Budget, op. cit.: 11-12. 



540 



1 . 1: is learr.ir.g zt.zz. by doing . . . . 

1. It is b-ilcitj: at interdisciplinary staff . Ftcti a ::::;;;::-=: staff larjelv 
c: rersr-s viz':. bac'Kgrc urcs it. acocctcir.E ace :::'.;::::- = : a.iitir.g. :.-.: is 
shifting t: a staff safe uc cf nar.v disciplines — incl-ding iz.zzz.eezs, econo- 
mists, mathematicians, statisticians, actuaries, systems analysts, persons 
vith academic or experience backgrounds in business or public administration, 
and a sprinkling of exper-- in other fields. Over 20 percent of Che profes- 
sional staff is composed of persons trained in disciplines other than account- 
ing and this proportion is growing. 

3. It makes extensive use of consultants in various fields to provide expert 
advice and assistance. 

U. It contracts vork out on occasion. 

5. It conducts extensive training rrc;r=-x= and holds special Srr-'-.ars or: orcgran 
evaluation in different fields. ...Seminars have been held on health, wel- 
fare, education, at: lav en: croEnenc assistance. 

6. It takes advantage of on— going or completed analytical and evaluation 
ycrk cf ether "ederal agencies and outside ergsniz-atiens , such as the Verse 
Institute and the Brookings Ins titration. It is developing a guide to pro gra m 

evaluation studies .... 

It r.as asssnrlsi at acTiscrv canel cf a::_: 2'. =~ = zezLS atd tr:gra- analysis 
specialists vho are expert in numerous fields of government operations and 

eval-atioc. , "- 

3. Illustrations ?rrvided by the 1-Al cf Its ?rcgran Evaluation Activities . 
Assistant Comptroller General Morse has identified examples of different —as :f 
program eval.acurs :-e age-.: -■ has t>erf crmed. "-/ One cf tcese is inserted below to 
illustrate the agency's conception of its activities in this area: 

Improvement of definition of a programs goals and :'c; actives 

Title I of the Elementary and Secondary Education Act of 1965 authorized 
Federal financial assistance to local educational agencies for programs to help 
meet special educational needs of deprived children in areas vith high concen- 
trations of children from low- income families. The Federal Office of Education 
required local agencies to establish measurable objectives and related pro- 
cedures to evaluate the success of their projects. GAO's audit of this program 
in Illinois showed weak performance on this score. Objectives stated were 



71 / Horse, Ellsworth B. Jr. The Auditor Takes a Program Evaluation. The Federal 
Accountant, v. XJJI, no. 2, June 1973: b-~ . Emphasis in original. 

22.1 Most GAD reports are public information. In addition short abstracts are 
published in annual GAO reports. 



541 



generally vague (e.g., build a varied vocabulary) and were not stated in measur- 
able terms by types and degree of changes sought (e.g. rate of increase in vocabu- 
lary). 73_/ 

4. Other GAP Program Evaluation Activities. GAO has also published two reports 
which deal more directly with improving the process of conducting program evaluations. 
These, described in detail in other sections of this report, consist of (1) a list 

of legislative requirements for program evaluation, including steps the Congress 
needs to take to state program objectives in a form compatible with evaluation 
capabilities and (2) an inventory of agency information systems which include program 
evaluation reports to support better dissemination of program evaluation reports to 
other researchers and to other agency administrators. The agency is now preparing 
a guidebook of recommended standards for program evaluation. It will be published in 
1975. 

5. Activities and Legislation in Support of Giving GAO More Responsibility for 
Program Evaluation. The General Accounting Office's program evaluation activities 
seem to have Increased considerably since passage of the Legislative Reorganization 
Act and the Budget and Impoundment Control Act. Several factors might be considered 
in assessing these expanded responsibilities and functions. Probably the most 
important are: 

-psychological, organizational, and methodological factors which surround the 
evident transition in the agency from considering evaluations as primarily easy-to- 
measure auditing activities to conceiving of evaluations as harder-to-measure achieve- 
ment, impact and performance activities; 

-steps the agency is taking to enlarge its staff to perform program evaluations 
and to enlarge its staff to include more social scientists to provide the inter- 
disciplinary mix required for program evaluation; and 



73 / Morse, op. cit. 



542 



CRS - 40 

-the time lag between establishment of new legislative authority and the estab- 
lishment of effective operational procedures to meet new requirements. 

As noted above, under the terms of the Congressional Budget and Impoundment Act, 
passed June 21, 1974, GAO was gi r 'en additional authority to conduct program evaluation 
and to engage in other activities in support of expanding legislative branch program 
evaluation capabilities. The agency is authorized to establish an Office of Program 
Review and Evaluation. Among its functions are the following: 

-to review and evaluate Government programs and activities, at the initiative of 
the Comptroller General or at the request of either House or Committees; 

-to assist committees in developing statements of legislative objectives and 
goals and methods for assessing and reporting actual program performance in relation 
to such legislative objectives and goals; and 

-to recommend methods of assessment, information which should be reported, respon- 
sibility for reporting, frequency of reports, and feasibility of pilot testing; and 

-to review the program evaluation reports and activities of Federal agencies. 74 / 



74 / Congressional Budget and Impoundment Control Act, Title VII, op. cit, 



543 



PART IV. THE PROCUREMENT AND CONDUCT OF 
PROGRAM EVALUATION RESEARCH 

In the most general sense, program evaluation is a form of applied social 
science research which uses the interdisciplinary resources and techniques of economics, 
psychology, sociology, and political science. Some program evaluation research is 
performed in-house by social scientists on an agency's staff. These evaluations 
generally are of two types: (1) auditing activities to assess the allocation and use 
of program funds; and (2) critical syntheses of the findings of the evaluation studies 
performed for the agency to generate recommendations for improving the administration 
and operation of a program or to determine whether to modify or terminate the programs. 

Evaluations of program performance or of the effectiveness and impacts of a 
particular program intervention strategy are performed generally on an extramural 
basis by social scientists or interdisciplinary program evaluation research teams 
funded by the agency. The Office of Management and Budget has defined four general 
types of evaluation research, delineated on the basis of their purpose and pre- 
dominant methodology. These generally correspond to the categories used by Wholey, 
et. al. in their landmark study on this topic. 75 / The types are substantive impact 
evaluations; relative effectiveness evaluations; process or management evaluations; 
and project evaluations. The OMB definitions of these types of evaluation are given 
below: 

Substantive impact evaluations attempt to measure the impact which federal 
programs have upon their stated objectives. This type of evaluation seeks to 
determine what the program, accomplishes, how these accomplishments compare to 
their intended purposes, and their costs. The purpose of such evaluations is 
primarily to provide information for use in major policy formulation. 

Relative effectiveness evaluations seek to compare the effectiveness of 
two or more major program strategies or approaches in attaining ultimate objec- 
tives within a national program. These studies are designed to help policy 



75/ Wholey, Joseph S., et al. Federal Evaluation Policy: Analyzing the Effects of 
Public Programs. Washington, D.C., The Urban Institute [1970] chap. 2. 



544 



officials and program managers select the most effective mix of services to 
maximize programs' total impact, such as the mix of skill training, remedial 
education, and job search assistance in a manpower program. However, these 
studies do not necessarily measure the impact of the total program in absolute 
terms on its objectives. 

Process or management evaluations are designed to measure the operating 
efficiency of national program. They are intended principally to help program 
managers achieve the most efficient deployment of available resources, rather 
than help policy officials arrive at major decisions affecting the scope and 
focus of the national programs. 

Project evaluations are directed to individual, locally based projects which 
are components of a national program, regarding the impact or efficiency of the 
total national program. Project evaluations may entail any of the three pre- 
ceding types (substantive impact, relative effectiveness, or process evaluation) 
as well as project rating comparing the effectiveness of one or more individual 
projects against others. j(J 

Four types of research performer are used to perform extramural evaluation re- 
search activities. The first two are: (1) social scientists supported by grant or 
contract to a non-profit making institution; and (2) social scientists at federally 
funded research and development centers, which are R and D performing organizations 
usually created and exclusively or substantially funded by one or more Federal agencies 
to provide the agency with a centralized R and D capacity needed to develop and 
evaluate its programs. Typical of these are the Urban Institute and the Institute 
for Poverty Research at the University of Wisconsin, whose major functions were to 
supply expert information and advice on urban and poverty programs, respectively. A 
third type of performer consists of university-based social scientists. The fourth 
type of performer is social scientists or interdisciplinary teams (including operations 
researchers and systems analysts, etc.) employed by profit-making research or con- 
sulting firms working under contracts or gr«nt6 awarded on a competitive or non- 
competitive basis by an agency. 

A. Issues Relating to the Funding of Program Evaluation Research 

1. The Lack of Appropriate Information. The executive branch does not maintain 



76/ Lewis and Zarb, op. cit.: 309. 



545 



CRS - 43 

an inventory describing precisely how much money has been obligated for program 
evaluation research. Frequently there are discrepancies in the data systems used. 
For instance, Laurence E. Lynn, Jr., former Assistant Secretary for Planning and 
Evaluation, Department of Health, Education and Welfare, noted in 1972 that "The 
Department has approximately 125 full-time professional evaluators and 40 to 50 
million dollars available for evaluation functions. "77/ According to a report cited 
by Albert D. Biderman and Laure M. Sharp, the costs for total HEW evaluation research 
amounted to $29.6 million in 1971.28/ 

Annual Budget Appendix volumes sometimes report an entry for program evaluation. 
However, this item is not reported for all social programs listed in the Catalog of 
Federal Domestic Assistance. For instance, the Fiscal Year 1974 budget appendix il- 
lustrates these discrepancies for three agencies which have large program evaluation 
research obligations: (1) Health Services and Mental Health Administration, lists an 
item for research, but not program evaluation; (p. 379); (2) Law Enforcement Assis- 
tance Administration lists have no entries for either research or program evaluation 
(p. 627); (3) Manpower Administration, Department of Labor, lists an item entitled 
Planning, evaluation, and research, (p. 631). 

Biderman and Sharp attribute the dearth of information on the funding of program 

evaluation research to the fact that existing scientific research reporting categories, 

including those used by NSF, do not Include the classification "program evaluation 

research"; and there are no well-established mechanisms within agencies to obtain data 

on the funding of such research: 

All available statistical data on the organization, financing, personnel 
and functions of research utilize categories that have no close correspondence 



77/ Lynn, Laurence E. Notes from HEW. Evaluation, Fall, 1972: 26. 

78 / Citing: James G. Abert. Cost and Cost-Effectiveness. Paper delivered at Inter- 
national Federation of Operational Research Societies International Cost-Effec- 
tiveness Conference, Washington, D.C., April 12-15, 1971. (Unpublished.) From: 
Biderman, Albert D. and Laure M. Sharp. The Competitive Evaluation Research 
Industry. Washington, D.C., Bureau of Social Science Research, Inc. [1972] p. 21. 



546 



with the world of SPER.* The primary sources of data on American research 
activities are oriented to the academic science system or to medical and 
technological research and development. SPER falls neatly into none of these. 
That most used source for charting trends in American research activity, the 
National Science Foundation's series. Federal Funds for Science , is of scant 
use for charting trends in ^PER. Even if cross-tabulations of data of this 
series were available by fi CJ .d of science, funding agency, and research per- 
former, which they are not, we would have no way of knowing which table cells 
would include SPER. 

While, by definition, all SPER is social research, not all of it is reported 
as social science research and, since it is often supported from operational 
program funds, an unknown proportion of the total is not reported as science 
or research at all. Further, allocations to evaluation work are frequently made 
by the executors of federally-financed action programs or projects — state, local 
or private agencies which may be layers removed from the point at which book- 
keeping entries are made that are reflected in NSF's Federal Funds for Science . 

Although there is considerable literature arising from interests in govern- 
ment contracting, including studies of R & D procurements ( Report to the 
President . . . ., 1962), studies have not focused on what until recently has 
been that relatively tiny part of all procurement which involves social scientific 
or related work. Other than those dealing with contracting for developments of 
defense weapon systems, the major studies have dealt primarily with the National 
Aeronautics and Space Administration and the Atomic Energy Commission (Senate 
Committee on Labor and Public Welfare, 1965; Orlans, 1967). When government 
financing of social scientific work has been the object of attention of studies 
(and data series), the preoccupations have been almost exclusively with the 
forms of financing which matter most for institutions of higher education and, 
secondarily, for the academically-oriented institutes. The types of organi- 
zations which perform most of the work of SPER have received relatively little 
consideration. 

In the recent large survey to examine the state and future of the Social 
and Behavioral Sciences (Behavioral and Social Sciences Study Committee of the 
National Academy of Sciences and the Social Science Research Council, 1969), for 
example, a comprehensive questionnaire survey was carried out of the research 
finances of academic departments, professional schools and university institutes 
and laboratories, but only a small pilot study in one city — scarcely analysed 
or reported — was performed of social scientists in all other institutions.^ 

The Survey Committee contracted with the Bureau of Social Science Research in 
Washington, D.C. for a questionnaire survey of university activities in the 
behavioral and social sciences which were reported in three chapters of this 
report (pp. 155-210). A small pilot survey was also undertaken by the committee 
in the Boston, Massachusetts area to study the employment of behavioral and 
social scientists in two disciplines (economics and psychology) outside the 
university and government; this was reported in two pages of the report (pp. 216- 
218).] Such studies as have been done of social scientists in non-academic 
employment (Radon, 1970; Marcson, 1966) have involved small samples including 
many persons working in non-research roles, so that it would be impossible to 
segregate out for analysis those active in particular research pursuits, such 
as SPER._7_o7 
* Social Program Evaluation Research 



79 / The Report on Survey of Federal Program Evaluation Practices , conducted by the 
staff of Senator William V. Roth, Jr., May 1972, illustrates the difficulties in 
trying to obtain comparable figures from Federal agencies even when a well- 
formulated questionnaire is used. Quotation from Biderman and Sharp, op. cit. p. 16-18. 



547 



B. Difficulties of Obta ining, Information on Types of Perform ers of Program Evaluation 
Research 

One of the most important implications of the absence of information is that it 
is impossible to provide precise data on the characteristics of performer, that is in- 
house, or extramural (either profit-making or non-profit-making research organization). 
A number of attempts have been made; they have involved in-depth surveys or collation 
of scattered in-house report forms. 

Most of these studies indicate that the bulk of the largest awards for evaluation 
are given on a contract, rather than a grant basis, to profit-making or non-profit- 
making research organizations rather than to university based social scientists. 80/ 
This finding should be weighed against studies which indicate that evaluation studies 
are performed best by social scientists at universities funded on grant, rather than 
a contract basis. 81/ 

For example, Abert reports that about 45 percent of the funds HEW awarded in 
1971 for program evaluation went to profit-making research organizations. The 
recipients of other funds are broken down as follows: not-for profit firms, 29 per- 
cent; universities, 21 percent, state and Federal government, 4 percent; and individual 
consultants, 1 percent. 82/ 

In the Spring of 1974, the General Accounting Office conducted a telephone 
survey of federally funded program evaluation on behalf of the Joint Committee on 
Congressional Operations. The results follow: 



80/ Biderman and Sharp, op. cit. 

81 / This issue was discussed for example at the "Dartmouth /OECD Seminar on Social 
Research and Public Policies," Dartmouth, September 13-15, 1974, especially in 
a presentation by Howard Freeman, based upon research he, Illene Bernstein and 
Patricia Riecker are doing for the Russell Sage Foundation. Draft report 
available from: Illene Bernstein, Department of Sociology, University of 
Indiana, and Dr. Freeman, director, Institute for the Social Sciences, UCLA. 
For a short summary of the report, see: Evaluation Revisited, Behavior Today, 
May 27, 1974: p. 148. 

8? / Abert in Biderman and Sharp, op. cit., p. 23. 



548 



The response to our survey, which are summarized in the following table were 
for the most part carefully researched and prepared by the agencies listed. A 
few of the agencies responded in writing-DOT, Agriculture, 0E0, VA, and the 
Special Action Office for Drug Abuse Prevention. Problems with the definitions 
of program evaluation were fewer than we expected. We believe the total estimate 
of $130 million budgeted for program evaluations in Fiscal Year 197A is fairly 
accurate for those agencies listed, given the limitations of a telephone survey. 

PRELIMINARY ESTIMATES OF FUNDS BUDGETED FOR PROGRAM EVALUATION, RESPONDING AGENCIES, FISCAL 

YEAR 1974 

(In thousands of dollars) 

Amounts budgeted for evaluation breakdown of 
total by- 



Agency 



Department of Health, Education and Welfare 50r, 000 10,000 60,000 

Department of Transportation 7,095 18,875 25,970 

Environmental Protection Agency 5,756 6,667 12,423 

Department of the Interior 3,800 7,584 11,384 

Department of Labor 8,900 567 9,467 

Department of Commerce 1,031 4,520 5,551 

Law Enforcement Assistance Administration 4,337 165 4,502 

Veterans' Administration 38 3,154 3,192 

Department of State (except AID) 13 3,091 3,104 

Department of Agriculture 2,000 1,075 3,075 

Office of Economic Opportunity 1,550 1,023 2,573 

Department of Housing and Urban Development 1,300 327 1,627 

Agency for International Development 528 1,098 1,626 

Special Action Office of Drug Abuse Prevention 869 653 1,522 

Total 87,217 58,799 146,016 

Percent of total 59.7 40.3 



Agencies unable to give a complete response in the time allotted for this 
preliminary telephone survey: Department of Defense, Department of Justice (Except 
LEAA), Department of Treasury, National Aeronautics and Space Administration. 83/ 

The agency did not give a breakdown by types of contract performer (that is profit 
or non-profit making). It seems evident that a continuing data system be established 
to collect information on federally-funded program evaluation and that it also cate- 
gorize the type of performer. 



8 ^ In U.S. Congress. Joint Committee on Congressional Operations. Congressional 

Research Support and Information Services: A Compendium of Materials. Washington, 
U.S. Govt. Print. Off. [May 1974] p. 290. 



549 



C. Specific Issues Relating to Federally Funded Research and Development Centers 
as Performers of Program Evaluation Research 

The Congress has not given in-depth examination to the conduct of social pro- 
gram evaluation research by federally funded research and development centers (FFRDC). 
It is difficult to generalize about these activities, except to say that some agencies 
have used them more than others; and that the use of such centers in the social policy 
area is probably declining. g^/ Some information on the pros and cons of the use of 
FFRDCs was provided in a 1971 report by the National Academy of Sciences. A special 
advisory committee reported to the Academy on the operations and implications of one 
such center, the Institute of Poverty Research, University of Wisconsin, sponsored by 
the Office of Economic Opportunity. The findings of this study are specific to this 
particular center but could characterize the activities of other centers. 

According to the Committee, policymakers should consider a number of issues 
which surround the conduct of program evaluation research of this nature: 

- the "danger of overselling the fruits of even well-conducted evaluations 
and and field experiments;" 

- the "strong partisan political currents" that interact with the conduct 
and use of reports prepared by such centers; 

- the need for replication of experiments initially conducted by the center; 

- a "division of responsibility for the conduct of an evaluation or ex- 
periment between two or more teams so that the knowledge of each would serve as 
a check against the oversight or misjudgments of the others"; 



The Office of Education made extensive use of such centers; they were phased 
out at the end of the fiscal year 1972. During the period 1971-1973 HEW, 
exclusive of the Office of Education utilized 17 such centers; during the 
period 1972-1974, the agency funded only 13 centers. (U.S. National Science 
Foundation, Federal Funds for Research, Development, and Other Scientific 
Activities, Washington. U.S. Govt. Print. Off., 1972, p. 46. (NSF 72-317); 
and U.S. National Science Foundation. Federal funds for Research, Development, 
and Other Scientific Activities. Washington, U.S. Govt. Print. Off., 
1974. pp. 51-52. (NSF 74-300). 



550 



- the need for "open access to the data for subsequent recalculation, criticism 
and evaluation by independent groups as required;" 

- the need to time appropriately General Accounting Office and executive 
agency investigations of a center's activities or research progress and preferably 
delay such reviews until an evaluation is complete, since "the threat of [un- 
timely] investigations of this sort... might discourage investigators of high 
caliber from involving themselves in such policy-related work;" 

- the need for policy makers at all levels of government to lower expec- 
tations about the utility of policy-oriented program evaluation research since 
"social science research is able to provide only very limited illumination; "85 / 
and 

- the need to consider a number of trade-offs between the advantages and 
disadvantages of research conducted by policy institutes and to provide the 
institutes with "long-term, relatively unconstrained funding" since Such insti- 
tutes cannot do "first-rate independent research [with] hand-to-mouth financing. "86/ 
The Commission on Government Procurement, created pursuant to P.L. 91-129, to 

study and recommend to the Congress methods "to promote the economy, efficiency, and 
effectiveness" of procurement by the executive branch, looked at the procurement of 
applied social science research and program evaluation services. The report, pre- 
sented to the Congress in December 1972, recommended that FFRDCs be organized and used 

to satisfy needs that cannot be satisfied effectively by other organizational re- 



85 ^ "While social science can help in the design and improvement of programs, good 
policy-making requires art, judgment, and weighing of conflicting values. It 
can never be reduced to [social] engineering in the familiar sense of the term." 
(Policy and Program Research In a University Setting: A Case Study. Report of 
the Advisory* Committee for Assessment of University Based Institutes for Research 
on Poverty, Division of Behavioral Sciences, National Research Council. [Wash- 
ington, D.C., National Academy of Sciences, 1971] p. 54.) 

86/ Ibid., pp. 47-55. 



551 



CRS - 49 

sources. Any proposal for a new FFRDC should be reviewed and approved by the agency 
head and special attention should be given to the method of termination, including 
ownership of assets, when the need for the FFRDC no longer exists. Existing FFRDCs 
should be evaluated by the agency head periodically (perhaps every three years) for 
continued needs. 87/ 

D. Issues Relating to Funding Mechanisms (Grants and Contracts) in Program Evaluation 
Research 

1. Issues Relating to Profit- Versus Non-Prof it-Making Performers . Albert D. 
Biderman and Laure M. Sharp, Bureau of Social Science Research, are conducting an in- 
depth study on the performance of program evaluation research by both profit- and not- 
for-profit research institutions. Their interim report, dealing primarily with profit- 
making research organizations was printed in 1972. Preliminary findings raise a number 
of issues resulting from the conduct of social program evaluation research performed 
by profit-making organizations. Generally they observe that competitive bidding for 
program evaluation research through the request for proposal (RFP) system is waste- 
ful of resources and deleterious to sound research since a large number of organi- 
zations go through the expensive process of writing unsuccessful proposals. They 
gave an example, for instance, of a $4 million procurement for which research 
organizations spent $1.3 million in writing proposals. 88/ 

They also cited the need to assess the disadvantages of the ways in which the 
RFP system tends to overly prespecify tasks or research methodology that a contractor 
should follow. 89/ On this same point the National Advisory Council on Education 



87 / Summary of the Report of the Commission on Government Procurement. (Not the 
Official Report.) Washington, U.S. Govt. Print. Off., December 1972. p. 46. 
For a detailed description of the Commission's findings on performers of re- 
search and development, see Chapter 3, pp. 13-23 in Report of the Commission 
on Government Procurement, Vol. 2 (Washington, U.S. Govt. Print. Off., 1973). 

88 / Biderman and Sharp, op. cit., pp. 38-39. 

89, 

/ Ibid., passim. 



71-867 O - 76 - 36 



552 



Professions Development reports: 

Within limited evaluation funding on the whole, funds for contract research 
are so overbudgeted and in-house staff positions and funds are so underbudgeted, 
that good, practical conceptalization and -monitoring of evaluation research is 
thwarted. In some cases even requests-for-proposals must be developed by out- 
side contractors before proposals for research can be called for. ...Requests- 
for-proposals have often attempted to specify the evaluation research design 
rather than the questions of or issues to be addressed, or the final character 
of the research evidence. 90/ 

Other issues raised by Biderman and Sharp and the Advisory Council are: the lack. 

of time agencies give to writing RFP's since many put out calls for contact program 

evaluation research only a few weeks before the fiscal year ends; 91/ monopolization 

of contracting by a few profit-making organizations; overutilization of systems 

engineers and systems analysts to perform program evaluation, often using imprecise 

social science tools and techniques; administrative deficiences in profit-making 

organizations; the economic efficiency and quality of the product obtained from the 

use of sole source contracts vs. the competitive bid proposal; and cost overruns in 

social R and D activities performed by not-for-profit organizations . 92 / 

E. Legislative Requirements to Mandate a Certain Percentage of Funding for Program 
Evaluation 

Another funding issue, which seems to warrant resolution, is that of legisla- 
tively mandating how much an agency should spend on program evaluation. There are 
two prevailing points of view on this issue* One is to continue the practice of 
having the Congress authorize a fixed percentage of appropriations of a program for 
evaluation purposes. A number of suggestions have been made about what percentage 
of funds should be allocated for this purpose. In an interview given shortly before 



90/ National Advisory Council on Education Professions Development. Search for 
Success: Toward Policy on Educational Evaluation. A report of the..., June 
1974. Washington, D.C., National Advisory Council on Education Professions 
Development, op. cit. p. 20. 

91/ Ibid., p. 20. 

92 / Biderman and Sharp, op. cit., passim. 



553 



he left office as Secretary of Health, Education and Welfare, Elliot Richardson 

suggested that an allocation of .5 percent of appropriations for a program would be 

adequate. 93/ In their study of Federal Evaluation Policy , Wholey et. al., suggested 

that the fixed percentage should vary depending on the substance of the program: 

Funding levels . A reasonable evaluation budget..., this study suggests is 
likely to range from 0.5 percent to 2 percent of the total program budget. 
[For programs which do not already include a fixed percentage for evaluation] 
it would appear reasonable ... to allocate up to 2 percent of manpower program 
funds, up to 0.5 percent of vocational educational funds, and up to 0.5 percent 
of Title I elementary and secondary education funds to program evaluation. 94/ 

In his interim report to congressional committee chairman, on "Legislative 

References to Evaluation, 1967-1972," August 11, 1972, the Comptroller General expressed 

another point of view — that legislation should not specify what percent of the 

appropriations should be devoted to evaluation. His statement follows: 

...[We] question whether a legislative requirement that a fixed percentage of 
program funds or a fixed dollar amount to be spent for program evaluation is 
wise, particularly where the agency may not be in a position to spend the money 
effectively. Rigid personnel regulation imposed by the executive branch can be 
a contributing factor; scarcity of trained personnel has undoubtedly been 
another factor. [In our studies of evaluation for the Congress] .. .we will give 
special attention to this [issue] in those cases where Congress has enacted 
explicit requirements for program evaluation. 95/ 

F. Alternative Institutions for the Conduct of Program Evaluation Research Under 
Federal Agency Auspices 

Although there are no precise figures on the composition of performers of social 
program evaluation research for the Government, it seems reasonable to estimate that 
about half of this research is performed by profit-making research groups or con- 
sulting firms, approximately 30 percent by non-profit institutions and 20 percent by 



93/ Conversational Contact .. .with Elliot L. Richardson. Evaluation, v. I, no. 1 
1972: 15. 

_9V Federal Evaluation Policy, op. cit., p. 82. 

95_/ Letter prefacing GAO Report B-161740, August 11, 1972. 



554 



universities. 96/ As noted throughout this report, many criticisms have been raised 
about the uneven quality of eva